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ABSTRACT 


As  computers  become  more  powerful,  they  are  used  to 
tackle  more  sophisticated  problems,  such  as  air  traffic 
control  and  hospital  patient  monitoring,  in  which 
reliability  really  matters.  Errors  in  systems  of  programs 
that  perform  these  tasks  may  have  serious  economic  and 
social  consequences.  Techniques  that  lead  to  the  production 
of  reliable  programs  can  both  reduce  the  risk  of  failure  and 
shorten  the  development  process. 

The  language  in  which  programs  are  written  can  have  a 
substantial  effect  on  their  reliability.  This  thesis 
discusses  the  design  of  programming  languages  to  enhance  the 
reliability  of  programs.  It  presents  several  design 
principles,  and  then  applies  them  to  particular  language 
constructs.  Since  one  can  not  logically  prove  the  validity 
of  such  design  principles,  empirical  evidence  is  needed  to 
support  or  discredit  them.  An  experiment  was  performed  to 
measure  the  effect  of  nine  specific  language  design 
decisions  in  one  context.  Analysis  of  the  frequency  and 
persistence  of  errors  shows  that  several  decisions  had  a 
significant  impact  on  reliability. 


CHAPTER  ONE 


INTRODUCTION 

^  E^oducing  reliable  programs 

Programs  that  do  not  fail  under  permissible  operating 
conditions  may  be  called  reliable  [Goodenough  1974].  Of 
course,  "failure”  and  "permissible”  must  be  defined  for  each 
program.  The  reliability  of  a  particular  program  is  a 
function  of  the  number  of  failures  and  the  impact  of  the 
failures.  Measures  of  reliability  that  have  been  proposed 
include  the  mean  time  between  failures  (the  probability  of  a 
failure) ,  the  mean  time  to  repair  a  failure  (the  impact  of  a 
failure) ,  and  the  availability  of  a  program  (both  the 
probability  and  impact  of  a  failure) . 

In  most  programs,  errors  are  located  and  repaired  during 
a  process  of  extensive  testing.  A  study  of  the  development 
of  several  large  systems  of  programs  (SAGE,  OS/360,  and 
software  for  the  Gemini  space  shots)  has  shown  that  45%  to 
50%  of  the  total  effort  was  expended  on  system  checkout  and 
test,  and  14%  to  17%  of  the  effort  went  into  coding  and 
auditing  (including  desk-checking  and  program  testing) 
[Boehm  1973].  For  the  sake  of  efficiency,  once  systems  of 
this  type  go  into  production,  use,  the  amount  of  run-time 
checking  is  usually  held  to  a  minimum.  Thus,  errors 
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remaining  in  such  a  system  often  go  undetected,  and  detected 
errors  generally  take  longer  to  repair. 

Techniques  that  lead  to  the  production  of  reliable 
programs  can  both  reduce  the  risk  of  failure  and  hasten  the 
development  process.  One  approach  to  the  attainment  of 
reliable  programs  is  the  development  of  programs  that  have  a 
very  high  probability  of  being  correct.  This  probability 
can  be  increased  in  two  ways:  by  reducing  the  probability  of 
programming  errors,  and  by  increasing  the  probability  of 
detecting  errors  that  are  made.  Appropriate  language  design 
can  contribute  to  both  of  these  goals. 

Any  programming  language  must  be  a  compromise  among  a 
number  of  conflicting  goals  (simplicity,  power,  generality, 
elegance,  machine-independence,  efficiency,  etc.). 
Generally,  the  reliability  of  programs  written  in  a 
programming  language  has  not  been  a  primary  language  design 
criterion.  Instead,  programming  languages  have  been  created 
primarily  to  provide  programmers  with  powerful  and 
convenient  constructs. 

Language  designers  have  come  to  realize  that  the  value 
of  programming  languages  comes  at  least  as  much  from  the 
errors  and  "cleverness”  that  they  prevent  as  from  the 
powerful  operations  that  they  make  convenient  [ Wirth  1971a]. 
Many  language  designers  have  proposed  replacing  error-prone 
language  features  (e.g.,  the  goto  [Dijkstra  1968],  global 
variables  [Wulf  and  Shaw  1973],  and  gointer  [Hoare  1973b]) 
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with  more  disciplined  structures.  While  many  persuasive 
arguments  have  been  presented  against  the  undisciplined 
language  features  and  in  favor  of  alternatives,  little 
empirical  evidence  has  been  offered  that  the  alternative 
language  features  achieve  the  desired  effect.  Furthermore, 
given  two  alternative  language  features,  each  of  which 
supposedly  enhances  the  reliability  of  programs  (e.g., 
expression  evaluation  rules  based  on  traditional  mathematics 
as  in  ALGOL  60  [Naur  1963],  or  based  on  uniformity  as  in  APL 
[IBM  1970a]),  how  should  a  language  designer  choose  between 
them? 

The  unaided  intuition  of  the  language  designer  is  not 
always  a  satisfactory  way  of  deciding  how  language  features 
affect  reliability.  Also,  it  seems  difficult  to  prove  (in 
the  mathematical  sense)  that  particular  language  features 
enhance  the  reliability  of  programs.  However,  empirical 
evidence  can  be  gathered  that  tends  to  confirm  or  refute 
reliability  claims. 

What  type  of  empirical  evidence  should  be  obtained?  It 
seems  obvious  that  the  most  directly  applicable  evidence 
requires  the  collection  of  errors  made  by  experienced 
programmers  writing  useful  programs  in  a  programming 
language  that  the  programmers  know  well.  Several  empirical 
studies  of  the  programming  process  have  been  performed,  but 
almost  all  of  their  methodologies  have  lacked  at  least  one 
of  these  requirements.  Studies  using  novice  programmers 


either  reject  the  use  of  an  existing  programming  language  as 
too  difficult  for  the  subjects  to  learn,  or  collect  evidence 
that  over-emphasizes  syntactic  errors.  Concentrating  only 
on  reading  programs  overlooks  the  interesting  questions  of 
selection  and  organization  of  constructs.  Language  features 
must  be  evaluated  in  the  context  in  which  they  are  used,  and 
creating  too  small  a  language  may  prevent  the  observation  of 
errors  resulting  from  the  interaction  of  language  features. 

1 . 2  Overview  of  the  research 

This  thesis  is  concerned  with  a  single  language  design 
goal:  the  identification  of  language  features  thar  enhance 

the  reliability  of  programs.  In  order  to  achieve  this  goal, 
four  language  design  principles  that  were  expected  to 
enhance  reliability  (useful  redundancy,  uniform  rules, 
similarities  to  natural  languages,  mathematics,  and  other 
programming  languages,  and  the  identification  and 
elimination  of  errors  characteristic  of  a  feature)  were 
exploited  in  order  to  redesign  a  programming  language.  Nine 
mutually  independent  features  of  the  language  were  altered 
to  reduce  the  frequency  and/or  persistence  of  the  errors 
attributable  to  the  features.  The  features  included: 

•  order  of  expression  evaluation 

«  assignment  symbol 

«  relation-connectors 

•  semicolon  as  statement  punctuation 

•  selection  statements 

•  iteration  statements 

•  delimiters  for  compound  statements 

•  inheritance  of  environment 

•  constants 
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In  designing  the  new  features,  an  attempt  was  made  to  ensure 
that  they  differed  from  the  original  features  only  in  ways 
that  were  expected  to  affect  reliability.  Two  groups  of 
reasonably  experienced  programmers,  one  working  in  each 
language,  were  observed  in  order  to  obtain  the  empirical 
evidence.  Statistically  significant  results  were  obtained 
which  demonstrated  that,  at  least  in  one  environment, 
several  of  the  language  design  decisions  affected 
reliability  significantly. 

In  summary,  this  thesis  contributes  to  the 
identification  of  language  features  that  enhance  reliability 
by : 

•  Presenting  both  general  design  principles  and  specific 
language  features  that  may  enhance  reliability 

•  Using  these  design  principles  and  seme  of  the  language 
features  in  the  design  of  a  programming  language 

•  Developing  a  methodology  to  gather  empirical  evidence 
with  which  to  support  or  discredit  language  design 
decisions 

•  Discussing  an  experiment  that  measured  the  effect  of 
nine  specific  language  design  decisions  in  one  context. 
Analysis  of  the  frequency  and  persistence  of  the  errors 
observed  in  the  experiment  shows  that  '  several  language 
design  decisions  had  a  significant  impact  on 
reliability. 
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1 . 3  The  contents  of  the  thesis 

Chapter  Two  outlines  both  general  design  principles  and 
particular  language  features  that  contribute  to  reliability. 
Observations  and  experiments  concerning  the  programming 
process  are  examined  as  ways  of  evaluating  the  effect  of 
different  language  features  on  reliability. 

Chapter  Three  begins  with  an  overview  of  the  methodology 
used  in  this  thesis,  then  discusses  the  initial  limitations 
placed  on  this  observation.  This  is  followed  by  a  detailed 
examination  of  the  steps  of  the  methodology,  including  their 
strengths  and  weaknesses. 

The  differences  between  the  two  programming  languages 
used  in  the  study  are  detailed  in  Chapter  Four.  The 
characteristic  errors  associated  with  each  of  the  altered 
language  features  are  discussed.  Evaluations  of  the 
expected  impact  of  the  language  features  on  reliability  are 
offered. 

Chapter  Five  contains  the  empirical  results  of  the 
thesis:  the  evaluation  of  the  combined  effect  on  reliability 
of  all  nine  features,  followed  by  a  discussion  of  the  impact 
of  the  individual  features.  Further  information  is 
presented  concerning  the  errors  attributable  to  language 
features  that  were  not  redesigned.  This  chapter  also 
contains  a  discussion  of  the  effect  of  the  skill  and 
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background  of  the  subjects  on  the  errors  observed,  and  of 
the  usefulness  of  some  diagnostic  aids. 

Chapter  Six  discusses  the  limitations  inherent  in  the 
methodology  and  the  implications  of  this  ivestigation  for 
language  design.  Topics  for  further  research  in  this  area 
are  proposed. 
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CHAPTEP  TWO 


BACKGROUND 


2 . 1  Introduction 

In  order  to  obtain  evidence  that  particular  language 
features  enhance  the  reliability  of  software,  it  is 
necessary  to  investigate  both  the  kinds  of  language  features 
that  might  contribute  to  this  goal  and  the  methods  for 
verifying  that  the  features  have  the  desired  effect. 
Section  2.2  discusses  ways  in  which  a  programming  language 
can  aid  programmers  in  producing  and  maintaining  software. 
Section  2.3  surveys  particular  language  features  that 
contribute  to  reliability,  in  the  categories  of  disciplined 
structures,  explicit  interfaces,  redundancy,  display  of 
program  text,  and  association  of  test  cases  wirh  modules. 
Section  2.4  presents  ways  in  which  flow  analysis  can 
contribute  to  reliability.  Section  2.5  describes  the  use  of 
redundant  information  by  compilers.  Section  2.6  outlines 
observations  and  experiments  that  have  been  used  to  obtain 
empirical  evidence  about  the  programming  process.  Finally, 
the  relation  of  this  thesis  to  previous  work  is  discussed. 

2 . 2  General  considerations 

Programming  languages  should  lead  to  confidence  in  the 
correctness  of  programs.  They  should  aid  the  programmer  not 
only  the  coding  process,  but  also  in  the  processes  of 
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(formally  or  informally)  proving  programs  correct  and  of 
maintaining  programs. 

Ultimately,  the  most  powerful  weapon  against  incorrect 
programs  lies  in  the  understanding  of  those  who  write  and 
check  them.  Programmers  must  master  their  languages  for  the 
purposes  of  both  writing  and  reading  programs.  Two  factors 
play  major  roles  in  a  programmer’s  ability  to  master  a 
language:  its  size  and  its  naturalness.  A  language  should 
contain  constructs  that  encourage  the  clearest  possible 
statement  of  the  programmer’s  intentions.  The  need  for  more 
constructs  must  be  balanced  against  the  need  to  keep  the 
language  simple  [ Hoare  1973a],  [Wirth  1974]. 

If  programmers  are  to  have  confidence  in  the  correctness 
of  their  programs,  then  the  language  that  programmers  use 
must  never  surprise  them  in  any  of  its  effects.  The  meaning 
of  features  in  a  language  should  be  precise,  and  correspond 
to  a  sense  of  "rightness”  from  experiences  with  natural 
languages,  mathematics,  and  other  programming  languages 
[Weinberg  1971].  (E.g.  ,  as  in  natural  languages,  multiple 

blanks  and  single  blanks  should  both  be  separators.) 
Language  designers  should  aim  for  a  language  in  which 
programmers  will  quickly  become  fluent,  without  continual 
reference  to  manuals  or  the  need  to  run  experimenral 
programs. 

A  language  should  contain  features  that  allow  violations 
of  a  programmer’s  intentions  to  be  detected  as  errors. 
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Furthermore,  this  detection  should  be  performed  as  early  as 
possible  (e.g.,  during  compilation)  both  to  speed  program 
development  and  to  free  the  programmer  from  complete 
dependence  on  running  a  sufficient  set  of  test  cases. 

A.nother  way  a  programmer  may  raise  his  level  of 
confidence  in  a  program  is  to  prove  it  correct.  Hoare  has 
suggested  that  formally  defining  the  semantics  of  a  language 
with  proof  rules  might  contribute  to  the  clarity  and 
reliability  of  programs  expressed  in  the  language  [Hoare 
1969].  For  example,  Hoare  has  found  a  number  of 
restrictions  on  procedure  invocations  that  facilitate  the 
construction  of  proofs  [Hoare  1971],  without  significantly 
burdening  the  programmer: 

1.  All  global  variables  accessed  by  the  called 
procedure  must  be  passed  in  a  parameter  list. 

2.  The  parameter  list  must  be  divided  into  two  lists, 
one  of  which  contains  the  arguments  that  may  be  altered 
by  the  called  procedure  and  the  other  of  which  contains 
the  arguments  whose  values  may  only  be  read  by  the 
called  procedure. 

3.  A  parameter  must  not  occur  both  in  the  list  of 
arguments  that  may  be  altered  by  the  called  procedure 
and  in  the  list  of  arguments  that  may  only  be  read  by 
the  called  procedure. 

4.  All  arguments  that  may  be  altered  must  be  distinct. 
Constructs  that  lead  to  difficult  proofs  are  less  likely  to 
be  used  correctly  than  constructs  that  yield  simple  proofs. 
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Unfortunately,  the  simplicity  with  which  proof  rules  can  be 
stated  does  not  always  correlate  with  simplicity  of  proof, 
(e.g.  ,  the  goto  [Clint  and  Hoare  1972  ]). 

Concern  with  reliability  can  not  stop  when  a  program  is 
"finished.”  Large  systems  never  achieve  stability;  they 
continually  undergo  maintenance  and  growth.  In  this 
process,  system  structure  degeneration  and  personnel 
turnover  decrease  familiarity  with  the  system.  Ultimately, 
the  required  maintenance  effort  increases  rapidly.  Eelady 
and  Lehman  studied  a  system  whose  maintenance  effort  rose 
exponentially  [Belady  and  Lehman  1971  ].  The  rapid  increase 
in  maintenance  effort  that  appears  in  aging  systems  can  be 
delayed  by  increased  communication  between  the  programmers 
implementing  a  system  and  those  maintaining  it.  It  is 
essential  that  a  programming  language  make  this 
communication  clear,  convenient,  and  unambiguous. 

2 . 3  Language  features  that  contribute  to  reliability 

2.3.1  Disciplined  structures 

It  should  no  longer  be  necessary  to  argue  the 
disadvantages  of  undisciplined  structures  such  as  goto  and 
pointer.  These  features  introduce  a  substantial  barrier  to 
the  systematic  communication  of  a  programmer® s  intent. 

However,  it  is  not  sufficient  to  merely  delete  these 
structures  from  the  language.  They  must  be  replaced  by 
disciplined  constructs  that  are  both  clear  and  efficient. 
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For  example,  several  alternatives  to  the  30to  have  been 
proposed.  Examinations  of  experimental  protocols  [Miller 
1973]  and  natural-language  commands  [ McKeeman  1974]  reveal 
that  the  specification  of  the  transfer  of  control  is  often 
implicit.  When  one  person  is  giving  directions  to  another 
(e.g. ,  in  a  recipe  for  cooking  spaghetti) ,  the  former  will 
rarely  say  something  like: 

”Now  go  to  step  three." 

More  likely,  the  recipe  will  be  given  in  the  following  form: 

"Put  the  spaghetti  in  boiling  water;  take  the  spaghetti 
out;  throw  it  against  a  wall.  If  the  spaghetti  sticks 
to  the  wall,  quit.  Otherwise,  start  all  over  again." 

Programming  requires  translating  a  "natural"  algorithm  like 

the  one  above  into  a  form  that  is  unnaturally  specific 

(i.e. ,  containing  explicit  transfers  of  control)  so  that  the 

algorithm  may  be  executed  by  a  machine.  Experience  with 

such  natural  language  descriptions  might  lead  tc  the 

replacement  of  the  2.2^2  constructs  involving  naming, 

sequencing,  repeating,  and  quitting  that  allow  programmers 

to  express  themselves  more  naturally. 

2.3.2  Explicit  interfaces 

Large  systems  must  be  written  and  understood  as  sets  of 
interconnected  programs.  Mastering  the  programs  separately 
is  relatively  easy,  compared  to  understanding  their 
interactions.  A  programming  language  should  cater  to  the 
construction  of  hierarchies  of  programs,  and  the  explicit 
statement  and  enforcement  of  their  interfaces.  If  each 
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program  is  to  be  small  (e.g.  ,  limited  to  one  page  of  source 
text)  ,  then  efficiency  requires  that  program  linkage 
overhead  at  run  time  should  be  low. 

2.3.2. 1  Hierarchies  of  procedures 

Many  languages  use  scope  rules  similar  to  those  of 
ALGOL  60  in  order  to  define  interfaces.  Data  items  declared 
in  outer  units  of  scope  are  automatically  available  in  any 
properly  nested  unit  of  scope.  This  rule  results  in  two 
types  of  errors.  Inadvertently  interposing  a  redeclaration 
of  a  global  name  in  an  inner  scope  causes  references  to  the 
name  in  the  inner  scope  to  be  bound  to  a  new  local  variable, 
rather  than  to  the  global  variable.  Failing  to  redeclare  a 
global  name  that  was  to  be  reused  to  identify  a  local 
variable,  either  through  omitting  an  entire  declaration  or 
misspelling  the  name  of  the  variable  in  the  new  declaration, 
causes  names  in  the  inner  scope  to  be  bound  to  the  global 
variable,  rather  than  to  the  new  local  variable.  These 
errors  could  be  avoided  simply  by  requiring  that  all 
variables  have  distinct  names.  This  is  an  impractical 
restriction  if  procedures  are  to  be  coded  without  complete 
knowledge  of  their  environment.  Wulf  and  Shaw  have 
enumerated  some  desirable  attributes  of  another  restrictive 
mechanism  [Wulf  and  Shaw  1973  ]. 

1 .  The  scope  of  a  name  should  not  automatically  be 

extended  to  inner  blocks. 


2.  The  right  to  access  a  name  should  be  granted  by 
mutual  agreement  between  creator  and  accessor. 

3.  Access  rights  to  a  structure  and  to  its  sub¬ 
structures  should  be  decoupled. 

4.  It  should  be  possible  to  distinguish  different  types 
of  access. 

5.  Data  definition,  name  access,  and  storage  allocation 
should  be  decoupled. 


Some  of  these  attributes  could  be  grafted  onto 
conventional  block  structure  in  the  following  manner: 
Programmers  must  explicitly  specify  the  access  rights  of 
each  inner  block  to  each  variable  and  procedure,  both  with 
the  item  whose  access  is  being  restricted,  and  with  the 
block  in  which  an  attempt  is  made  to  access  the  item.  For 
example,  a  variable  might  have  access  rights  that  permit 
instances  of  it  to  be  allocated  or  deallocated,  and  its 
value  to  be  read  and  written;  the  right  xo  invoke  a 
procedure  might  also  be  explicitly  specified. 
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In  the  following  example,  the  first  two  executable 
statements  of  procedure  B  would  be  valid.  However,  the 
third  statement  would  be  invalid  since  B  may  neither  alter 
the  value  of  X  nor  access  the  value  of  Y. 

procedure  A  known  to  E; 

declare  X  read  by;  B,C; 

Y  written  b^  B; 
declare  Z  read  by  C; 

procedure  E  knows  A ,  reads  X ,  writes  Y ; 
oce d ur e  C  reads  X , Z ; 

• 

end;  /*  of  procedure  C  */ 

Y :  =xT'’ 

call  A; 

X:=Y;  /*  this  is  detected  as  a  double  error  */ 

end;  /*  of  procedure  E  */ 
end;  /*  of  procedure  A 
2. 3. 2. 2  Hierarchies  of  data 

Current  research  suggests  that  data  structures  provide  a 
basis  for  hierarchical  modularization  that  is  at  least  as 
important  as  (and  less  well-understood  than)  that  given  by 
procedures.  SIMULA  67  classes  [Dahl  et  al.  1970]  are  the 
starting  point  for  several  new  language  designs 
incorporating  data  modules.  For  example,  a  cluster 
definition  in  CLU  [ Liskov  and  Zilles  1974]  contains  an 
interface  specification,  an  object  representation,  code  to 
create  objects,  and  operation  definitions.  The  interface 
specifies  the  name  of  the  cluster,  the  parameters  required 
to  create  an  instance  of  the  cluster,  and  a  list  of 
operations  the  programmer  may  perform  on  objects  of  the  type 
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implemented  by  the  cluster.  These  objects  are  viewed  as 
atomic  by  the  users  of  the  cluster.  The  representation  of 
the  object  (i.e. ,  its  storage  structure)  may  be  manipulated 
only  in  the  operations  within  the  cluster. 

In  the  following  example,  a  cluster  is  used  to  implement 
a  stack,  where  the  type  and  maximum  number  of  elements  are 
not  known  in  advance,  with  the  operations  Push  and  Pop. 
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The  object  representation  (rep)  of  the  stack  is  a  record 
containing  four  components:  Stack_top  which  contains  the 
position  of  the  top  element  in  the  stack,  Stack^limit  which 
indicates  the  maximum  number  of  elements  the  stack  may 
contain,  Element_type  which  indicates  the  type  of  the 
individual  elements,  and  Stack_array  which  holds  the  values 
pushed  on  the  stack. 


Stack:  cluster  (Stack_type : t^^e ,  Limit: integer) 
is  Push, Pop: 

rep  (Type__para meter :t_ype)  = 

(Stack  top; integer,  Stack_limit ; integer , 
Element_type: tppe , 

Stack_^  array:array  [  1  to  Limit]  cf  Type^parame ter)  ; 
create 

S;rep  (Stack_type) ; 

S.Stack_top  :=  0; 

S . Stack_limit  :=  Limit; 

S . Element_type  :=  Stack__type; 
return  S ; 
end ; 

Push : operation  (S:rep,  Value: S.Element_type) ; 
S.Stack_top  :=  S.Stack_top  +  1; 
if  S.Stack_top  >  S.Stacklirait  then  error ; 

S . Stack_array  (S. Stack_top)  :=  Value; 
end; 

Pop: operation  (S;rep)  returns  S . Flement_type ; 
if  S.Stack__top  =  0  then  error ; 

S.Stack_top  ;=  S.Stack_top  -  1; 
return  S. Stack_array (S. Stack_top+1)  ; 
end ; 

end  stack ; 
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A  declaration  of  an  object  of  the  type  inplemented  by  a 
cluster  causes  the  actual  parameters  of  the  declaration  to 
be  bound  to  the  formal  parameters  of  the  cluster  interface 
specification,  and  the  create  operation  to  be  executed.  The 
declaration: 

S: Stack  (integer,  100) 

creates  a  stack  S  of  at  most  one  hundred  integers .  The 
values  of  S.  Stack__top,  S.  Stack_li mit ,  and  S.Element_t  ype 
would  be  0,  100,  and  integer  respectively. 

Clusters  allow  programmers  to  write  programs  in  terms  of 
the  objects  and  structures  that  the  programmers  find  natural 
for  particular  problems.  They  also  collect  and  limit  the 


distribution 

of  information 

about  how  an 

object 

is 

represented. 

making  the 

resulting  programs 

eas ier 

to 

maintain  and 

prove  correct. 

The  proof  of  a 

program 

is 

divided  into  two  parts:  a  proof  that  the  cluster  implements 
the  type,  and  a  proof  that  the  program  ,  using  the  type  is 
correct. 

2.3.3  Redundancy 

Much  of  the  readability  of  high  level  languages  comes 
from  their  conciseness,  based  on  the  suppression  of 
irrelevant  detail  or  the  exploitation  of  context.  However, 
a  certain  amount  of  redundancy  is  needed  for  easy 
readability.  For  example,  in  a  "typeless”  language  (i.e.,  a 
language  with  neither  compile-time  nor  run-time  typing) , 
operators  must  convey  complete  information  (different 
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symbols  would  be  used  for  integer,  real,  and  complex 
addition).  In  a  language  with  types,  the  context  of  an 
operator  is  used  in  determining  its  semantics  (the  same 
symbol  could  be  used  for  all  additions  with  the  type  of  the 
addition  determined  by  its  operands). 

More  importantly,  all  error  detection  is  based  on  the 
availability  of  redundant  information.  However,  not  all 
redundancy  contributes  to  error  detection.  Some  forms  of 
redundancy  invite  errors  by  reguiring  that  duplicate 
information  be  provided.  For  example,  the  external 
attribute  in  PL/I  [IBM  1970b]  permits  separately  compiled 
procedures  to  share  variables.  However,  if  the  variables 
are  not  declared  identically  in  each  of  the  procedures, 
errors  may  occur  at  run  time.  The  linkage  editor  simply 
merges  all  external  variables  with  identical  names.  In 
contrast,  the  mechanism  for  separate  compilation  in  the 
Project  SUE  System  Language  [Clark  and  Horning  1971] 
eliminates  useless  redundancy  by  reguiring  that  only  a 
single  identifier  be  duplicated  to  permit  sharing  of 
variables  and  complete  type  checking. 

Another  example  of  useless  redundancy  that  invites 
errors  is  the  need  to  specify  the  name  of  a  variable  twice 
in  order  to  increment  it. 

Identifier  :=  Identifier  *  1; 

The  addition  of  redundant  operations  to  the  language  (such 
as  those  shown  below)  can  eliminate  this  source  of  errors. 
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increment  (Identifier) ; 

Identifier  :+  1; 

Any  form  of  redundancy  will  lengthen  the  program  next, 
increasing  the  opportunity  for  "clerical”  errors.  Thus, 
language  designers  should  be  careful  to  only  introduce 
redundancy  that  leads  to  the  detection  of  more  errors  than 
it  causes. 

To  be  effective,  redundancy  must  cause  errors  to 
transform  valid  programs  into  detectably  invalid  programs. 
Useful  forms  of  redundancy  exist  in  languages  with  type 
checking,  declarative  redundancy,  and  invariance  conditions 
or  assertions. 

The  examples  of  redundancy  offered  in  this  section  are 
very  intuitive.  Although  coding  theory  provides  a  formal 
measure  of  redundancy  for  finite-state  codes,  and  techniques 
for  generating  error-correcting  codes,  these  do  not  seem  to 
be  directly  applicable  to  programming  languages,  primarily 
because  programmers  can  not  be  expected  to  reliably  provide 
such  "low-level"  redundancy. 

2.3.  3.1  Ty2® 

The  type  attribute  in  declarations  is  an  effective  form 
of  redundancy,  since  the  context  of  each  appearance  of  a 
data  item  can  be  checked  against  its  declared  type.  Both 
"typeless"  languages  (e.g.,  BLISS  [Wulf  et  al.  1971]  and 
BCPL  [Hichards  1969  ])  and  languages  with  automatic  type 
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PL/I)  defeat  type  checking. 


while 


conversion  (e.g., 
languages  with  large  numbers  of 
Pascal  [Wirth  1971a])  enhance  it. 


incompatible  types 


(e. g. , 


Pointers  cause  additional  problems  [Hoare  1973b],  [Wirth 
1974].  By  introducing  the  type  pointer  and  forbidding 
arithmetic  operations  on  objects  with  this  type,  high-level 
languages  have  avoided  some  errors  common  to  the  use  of 
pointers  in  assembly  languages.  However,  in  PL/I,  pointers 
may  be  used  to  access  objects  whose  types  are  unknown.  The 
following  PL/I  declarations  specify  that  A  is  an  integer 
halfword  and  that  B  is  a  character  string  of  length  ten. 
They  also  implicitly  declare  P  and  Q  to  be  pointers. 

declare  A  fixed  binary  (15,0)  based  (P)  ; 

declare  B  character  (10)  based  (Q) ; 

In  the  following  sequence  of  statements,  the  pointer  P  is 
used  as  though  it  referenced  a  data  item  (B)  whose  type  and 
size  are  different  from  those  of  the  data  item  (A)  to  which 
it  actually  refers. 

allocate  A  set  (P)  ; 

P->B  =  *abcdefghi j ’ ; 

This  problem  can  be  eliminated  by  requiring  that  pointers  be 
declared  with  the  type  of  data  they  reference,  as  is  done  in 
Pascal,  the  Project  SUE  System  Language,  and  ALGOL  68  [  van 
Wijngaarden  et  al.  1969].  The  ALGOL  68  declarations  below 
specify  that  P  will  only  reference  integer  values  and  Q  will 
only  reference  character  strings  of  length  ten. 

ref  int  P; 

ref  [1:10]  char  Q ; 
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2 . 3 . 3 . 2  Declarative  redundancy 

The  declaration  of  further  information  about  variables 
(e.g. ,  indicating  that  rhey  are  restricted  to  a  subrange  of 
their  type  as  in  Pascal,  or  that  they  represent  values  with 
certain  units  and  dimensions  [ Hoare  1973a])  may  permit  the 
easy  detection  of  what  would  otherwise  be  subtle  "logical" 
errors.  Such  information  also  provides  valuable 
documentation,  and  may  enable  the  compiler  to  perform 
optimizations,  particularly  in  storage  allocation.  In 
general,  it  helps  to  explicitly  declare  information  that 
must  otherwise  be  inferred  by  examining  all  uses  of  a 
variable . 

A  language,  such  as  PL/I,  that  does  not  require 
declaration  of  data  items  will  treat  the  occurrence  of  a 
misspelled  reference  as  a  contextual  declaration  of  a  new 
variable.  Some  compilers  for  these  languages  (e.g.,  the 
PL/I  Optimizing  Compiler)  produce  compile-time  warnings 
about  uninitialized  variables.  A  diagnostic  compiler  for 
such  a  language  (e.g.,  PL/C  [Conway  and  Wilcox  1973  ])  may 
produce  a  warning  at  run  time  that  the  contextually  declared 
variable  has  not  been  initialized, 

A  language  with  reserved  words  and  mandatory  declaration 
of  identifiers  allows  the  detection  and  repair  of  many 
keypunching  and  spelling  mistakes  at  compile  time 
(particularly  if  programmers  use  long,  dissimilar  names) . 
Morgan  has  proposed  an  algorithm  for  repairing  spelling 
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errors  involving  single-character  errors  or  transposed 
characters  [Morgan  1970].  He  claims  that  eighty  per  cent  of 
the  spelling  errors  made  in  programming  are  correctable  by 
this  algorithm.  A  language  designer  who  intended  to  use 
this  algorithm  would  select  long,  dissimilar  reserved  words. 
For  example,  the  reserved  words  read  and  write  would  be 
preferred  to  get  and  £ut  for  input/output.  A  single 
character  replacement  error  in  get  or  gut  (e.g.,  gut  or  get) 
would  leave  the  algorithm  with  only  an  even  chance  of  making 
the  correct  repair. 

Another  language  feature  that  reduces  useless  redundancy 
is  the  declaration  of  constants.  Banning  all  "mystery 
numbers”  (i.e.,  integer  constants  greater  than  2)  outside  of 
declarations  for  named  constants  can  lead  to  more  readable 
programs.  All  expressions  containing  the  named  constant 
will  use  a  consistent  value.  Furthermore,  to  change  the 
value  of  a  constant  each  place  it  occurs  in  a  program,  only 
the  declaration  for  the  constant  need  be  altered.  Unlike 
initialized  variables,  inadvertent  attempts  to  assign  new 
values  to  constants  can  be  detected  as  errors. 

2. 3. 3. 3  Assertions  and  invariance  relations 

A  programmer  can  also  supply  redundant  information  about 
his  program  by  making  assertions  or  stating  invariance 
relations.  Assertions  are  true  or  false  at  a  particular 
point  in  a  program,  while  invariance  releations  hold  over 
several  statements  of  a  program  (e.g.,  a  procedure). 
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Dijkstra  has  used  the  following  example  to  illustrate  an 
invariance  condition  [Dijkstra  1972].  Assume  the  relation: 
0  <  R  <  Dd 

is  true  before  the  execution  of  the  statements: 

Dd  :=  Dd  /  2; 

if  Dd  <  R  do  R  :=  R  -  Dd; 

Does  the  execution  of  the  statements  leave  the  relation 
invariant?  Execution  of  the  first  statement  leaves  R 
unchanged,  but  Dd  is  halved  so  the  following  relation  will 
hold: 

0  <  R  <  2  «  Dd 

Now  we  distinguish  two  mutually  exclusive  cases.  If  Dd  <  R, 
the  relation  above  can  be  used  to  infer  that: 

Dd  <  R  <  2  *  Ed 

Furthermore,  the  statement  following  the  do  is  executed, 
decreasing  R  by  Dd.  Therefore  our  original  relation  holds. 
In  the  second  case  if  Dd  >  R,  the  statement  following  the  do 
is  skipped.  From  the  first  step  we  have  that: 

0  <  R  <2  ♦  Dd 

However  in  this  case  we  can  impose  the  stricter  relation 
that  R  <  Dd  to  obtain  the  original  condition.  Since  the 
relation  holds  through  both  paths  of  execurion,  it  is 
invariant  over  these  statements. 

Most  compilers  that  include  the  assertion  feature  (such 
as  those  for  the  Project  SUE  System  Language  and  ALGOL  W 
[ Satterthwaite  1972])  generate  code  to  evaluate  and  test  the 
assertions  at  run  time.  However,  program  verifiers  can 
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check  at  (or  before)  compile  time  that  the  source  statements 
and  assertions  are  consistent,  using  a  mechanical  theorem 
prover  [Igarishi  et  al.  1973]. 

2.3.  3.4  Program  text 

In  the  maintenance  process  in  particular,  it  is  vital 
that  the  text  of  a  program  exhibit  the  logical  structure 
that  guided  its  construction.  If  structured  programs  are 
built  according  to  some  programming  methodology  (e.g., 
stepwise  refinement  [ Wirth  1971b],  modular  decomposition 
[  Parnas  1972c],  or  action  clusters  [Naur  1969  ]),  the 
language  should  not  only  support  this  process,  but  record 
it,  so  that  future  readers  of  the  program  can  follow  the 
series  of  decisions  that  led  to  the  program's  final  form. 
(Perhaps  a  more  realistic  goal  is  to  record  the  series  of 
decisions  that  would  have  been  taken  had  no  mistakes  been 
made  in  the  construction  process,  although  documentation  of 
corrected  mistakes  can  also  be  educational.)  Procedures  and 
clusters  may  often  be  natural  units  for  program 
construction.  However,  unless  their  overhead  is  low,  there 
may  be  some  temptation  to  gain  efficiency  by  removing  them 
from  the  final  form  of  the  program.  Source-program 
structure  and  run-time  efficiency  can  be  combined  if  the 
compiler  itself  has  the  capability  of  selectively  expanding 
procedure  and  cluster  bodies  ”in  line"  at  points  of 
invocation . 
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Even  so  simple  a  matter  as  the  formatting  of  the  program 
text  on  the  page  can  have  a  profound  influence  on  the 
reader's  ability  to  comprehend  it  readily.  Current 
empirical  studies  on  the  "psychological  complexity"  of 
programs  are  seeking  to  guantify  this  effect  and  to 
objectively  differentiate  between  good  and  bad  paragraphing 
styles  [Weissman  1974].  It  seems  clear  that  difficulty  in 
paragraphing  is  a  symptom  of  more  basic  language  flaws 
[Gordon  1975].  Many  compilers  now  either  automatically 
paragraph  source  listings  or  check  for  consistency  between  a 
program's  indentation  and  its  syntactic  structure. 

2 . 3 . 3 . 5  Test  cases 

Every  program  unit  should  have  a  set  of  test  cases 
permanently  associated  with  it,  for  original  validation,  and 
re-validation  after  every  change  [Poole  1973].  It  is 
perhaps  debatable  whether  we  should  incorporate  these  in  the 
language  (as  a  kind  of  generalized  assertion)  or  relegate 
them  to  a  support  system,  but  their  desirability  is  clear. 

It  has  been  suggested  that  flow  analysis  could  be  used 
to  detect  errors  in  statements  that  are  syntactically 
correct,  but  that  make  no  semantic  sense  [Elspas  et  al. 
1971].  In  the  following  example,  if  nhe  value  of  I  is  ever 
zero,  the  program  will  enter  an  infinite  loop. 

Lab:  if  I  =  0  then  goto  Lab; 


26 


Flow  analysis  can  discover  sections  of  code  that  can 
never  be  reached  and  values  of  variables  that  are  dead.  In 
the  following  example,  the  first  value  of  A  is  dead  because 
its  value  is  not  subsequently  fetched  before  A  is 
reassigned;  and  no  transfers  of  control,  or  labels,  that  are 
the  target  of  transfers,  appear  between  the  assignments  to 
A. 

A  :=  B*2; 

• 

.  no  references  to  A, 

.  transfers  of  control,  or  labels 

A  :=  B-1; 

The  programmer  probably  intended  to  use  the  first  value  of 
A,  but  omitted  a  statement.  A  warning  could  be  printed  to 
inform  the  programmer  that  a  value  of  a  variable  is  not 
used. 

2 . 5  C ompil^s  and  their  use  of  redundancy 

There  have  been  several  languages  and  compilers  whose 
intent  was  to  provide  programmers  with  improved  diagnostics. 
Among  the  better  known  of  these  languages  and  compilers  are 
CORC  [Freeman  1964  ],  DITRAN  [Moulton  and  Muller  1  967  ],  PL/C, 
and  ALGOL  W.  Many  of  the  diagnostic  features  of  these 
compilers  result  from  the  use  of  redundant  information. 
(E.g. ,  a  variable,  which  has  not  been  the  target  of  an 
assignment,  does  not  have  the  value  contained  in  the 
location  to  which  the  variable  was  bound,  but  a  special 
’’undefined”  value.) 
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CORC,  the  Cornell  Computing  Language,  issues  messages  at 
compile  time  for  variables  that  are  declared,  but  not 
subsequently  referenced,  and  for  statement  labels  rhat  are 
not  the  target  of  any  transfer  statements.  During  the 
execution  of  the  program,  checks  are  performed  for  the 
validity  of  arguments  of  functions  and  iterative  statements, 
and  for  subscripts  lying  within  the  declared  bounds  of  the 
arrays  they  reference.  An  independent  check  is  made  of  the 
condition  controlling  an  iteration  statement  to  determine 
the  number  of  times  the  iteration  should  be  performed.  This 
allows  detection  of  any  interference  with  the  components  of 
the  condition  that  might  cause  an  infinite  loop. 

DITRAN,  Diagnostic  FORTRAN,  contains  checks  for  computed 
GOTOs,  library  function  parameters,  and  subscript  bounds. 
All  references  to  the  values  of  variables  are  also  checked 
to  see  if  a  value  is  undefined.  A  variable  has  the  value 
undefined  if  no  statement  executed  has  assigned  a  value  to 
the  variable,  or  if  the  variable  has  become  undefined 
because  of  the  execution  of  other  statements  in  the  program 
(e.g. ,  in  some  implementations  of  FORTRAN,  the  value  of  the 
index  of  a  do  loop  is  undefined  if  the  loop  terminates 
normally)  . 

PL/C  is  a  subset  of  PL/I,  with  improved  diagnostic 
facilities,  developed  at  Cornell.  PL/C  contains  many  checks 
similar  to  those  mentioned  above,  such  as  substring  or  array 
index  out  of  bounds.  To  detect  uninitialized  variables. 
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each  variable  is  initialized  to  a  null  value  when  space  is 
allocated  for  it,  and  the  value  of  the  variable  is  tested  on 
each  reference  to  it, 

A.LGOL  W  is  an  extension  of  A.LGOL  60  with  list  processing 
and  additional  data  types.  The  compiler  provides  execution¬ 
time  checks  for  incorrect  substring  or  array  indexing 
operations,  invalid  values  of  selection  expressions  in  case 
statements,  assignments  to  name  parameters,  and  the 
correspondence  between  the  number  of  actual  and  the  number 
of  formal  parameters.  There  is  also  an  assert  statement. 

Pascal  and  the  Project  SUE  System  Language  and  their 
respective  compilers  make  use  of  some  of  the  additional 
redundancy  outlined  in  previous  sections.  Both  languages 
and  compilers  possess  subrange  types  and  the  ability  to 
perform  range  checking,  typed  pointers,  and  forms  of  the 
case  statement  that  provide  for  selection  of  statements  by 
name  rather  than  by  position.  In  addition,  Pascal  supports 
iterative  statements  that  discourage  alteration  of  the  value 
of  either  the  control  variable  or  the  upper  limit  of  the 
loop.  As  previously  mentioned,  the  Project  SUE  System 
Language  contains  an  assert  statement. 

2 . 6  Gathering  empirical  evidence 

In  order  to  gather  empirical  evidence,  it  is  possible  to 
observe  programmers  at  work  and  to  examine  the  programs  they 
create.  Experiments  can  be  designed  to  investigate  portions 
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of  the  programming  process,  and  to  reduce  the  bulk  of  data 
that  observation  yields.  However,  experiments  also  have 
drawbacks  [Weinberg  1971],  The  behavior  of  the  subjects  in 
an  experiment  may  be  so  constrained  that  effects  that  would 
be  observed  under  natural  working  conditions  never  appear. 


2.6.1  Observations 


The  implementers  of  DITRAN  summarized  the  e 
in  three  months  of  usage  of  the  compiler  by  sever 
students  in  an  introductory  programming  course, 
than  five  thousand  programs  examined,  36% 
syntactic  errors  and  33%  contained  execut 
(excluding  exceeding  the  execution  time 
Approximately  one  quarter  of  the  compile-t 
occurred  in  arithmetic  assignments,  and  another 
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changes  were  uniformly  distributed  up  to  10  statements.  The 
number  of  changes  declined  exponentially  with  the  run  number 
of  the  program.  However,  the  proportion  of  changes 
involving  only  one  statement  did  not  increase  in  later  runs. 
The  types  of  statements  most  often  altered  were 
declarations,  selections,  outputs,  and  assignments.  Of 
course,  this  technique  is  very  crude;  it  is  incapable  of 
dealing  with  sophisticated  programming  errors,  or 
distinguishing  them  from  changes  in  programming  strategy. 

In  a  study  of  318  programs  written  in  FORTH^iN,  PL/I,  and 
assembly  language  over  a  five  day  period  by  ISS  users  at  an 
IBM  research  center,  Boies  and  Gould  found  that  only  12%  to 
17%  of  the  programs  submitted  to  the  language  processors 
contained  syntactic  errors  [Boies  and  Gould  1972].  The 
percentage  of  runs  containing  syntactic  errors  is  only  half 
that  observed  by  the  implementers  of  DITRAN.  Since  so  few 
runs  contained  syntactic  errors,  the  experimenters  concluded 
that  substantial  gains  in  programming  productivity  would  not 
result  from  improved  syntactic  error  identification.  They 
also  observed  that  the  utility  of  systems  for  the  repair  of 
syntactic  errors  must  be  evaluated  in  terms  of  the 
probability  of  such  errors  occurring. 

i 

Youngs  studied  the  errors  made  in  programming  by 
observing  programmers  working  in  high-level  languages 
[Youngs  1969  ],  [Youngs  1974].  Thirty  beginning  programmers 
and  twelve  advanced  programmers  each  coded  one  or  two  of 
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nine  problems.  The  solutions  to  the  problems  ranged  from  20 
to  80  statements.  Youngs  required  the  participants  to 
submit  run  logs  and  all  the  computer  output  from  the 
problems.  The  run  logs  contained  information  concerning  the 
preparation  time  for  the  run  (excluding  keypunching) , 
descriptions  of  the  errors,  and  the  modifications  made  to 
correct  the  errors.  The  experimenter  classified  each  error 
according  to  his  coding  scheme  and  estimated  the  number  of 
constructions  in  the  program  that  were  identical  to  the 
construction  containing  the  error.  Over  a  quarter  of  the 
1258  errors  observed  occurred  in  assignment  statements. 
Conditional  branches  contained  only  five  per  cent  of  the 
observed  errors,  but  were  five  times  more  likely  to  contain 
errors  than  other  statements. 


Few  language  design  conclusions  were  drawn  from  nhis 
study.  For  example,  the  iterative  mechanism  of  PL/I  was 
judged  to  be  more  likely  to  contain  errors  than  that  of 
FORTRAN.  Youngs  speculated  that  the  FORTRAN  requirement 
that  each  do  statement  carry  with  it  the  label  of  its 
terminating  line  prevented  the  subtle  error  of  mismatching 
do-end  keywords  in  PL/I.  Youngs  stated  that  the  validity  of 
his  study  would  be  most  effectively  demonstrated  by  showing 
that  the  data  collected  was  indeed  helpful  in  improving 
programming  languages  and  that  such  a  demonstration  had  been 
postponed . 
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One  language  design  effort  identified  some  errors 
typical  to  particular  constructs  [Ichbiah  and  Rissen  1971]. 
These  errors,  called  characteristic  errors,  are  found  in 
constructs  that  implicitly  involve  the  notion  of  order,  deal 
with  global  data,  or  rely  on  information  known  only  at  run 
time. 


For  example,  the  case  statement,  as  originally  proposed 
by  Hoare  [ Hoare  1964],  is  susceptible  to  two  types  of 
errors.  The  first  type  of  error  involves  the  omission, 
addition,  or  rearrangement  of  the  clauses.  A  second  type  of 
error  occurs  when  the  index  expression  contains  a  value  less 
than  ,the  index  of  the  first  clause  or  greater  than  the  index 
of  the  last  clause. 

declare  Current  fixed; 

case  Current  of 

SI; 

S2; 

• 

Sn ; 

end; 

Therefore,  the  construct  was  redesigned  to  incorporate  some 
useful  redundant  information.  The  index  of  the  case  clause 
was  constrained  to  be  an  expression  with  a  subrange  type. 
One  of  the  values  from  the  range  could  be  associated  with 
each  statement  in  the  case  clause,  which  permits  selection 
of  a  statement  by  name  rather  than  by  position.  This  makes 
the  order  of  statements  within  the  case  clause  irrelevant 
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and  permits  checking  for  omitted  or  extraneous  clauses,  as 
well  as  for  the  value  of  the  index  expression  being  our  of 
range.  In  the  following  example,  the  variables  of  rype  Unit 
are  limited  to  the  values  Reader,  Printer,  Punch,  Tape,  and 
Disk.  The  variable  Current  is  declared  to  be  of  type  Unit 
and  the  case  statement  specifies  that  Current  is  the  index 
and  Unit  the  appropriate  range  of  values. 

type  Unit  =  (Reader , Printer, Punch, Tape, Disk)  ; 

declare  Unit  (Current)  ; 

• 

case  Current: Unit  of 

Reader :  . . . ; 

Punch  :  . .  .  ; 

Tape,  Disk :  . . . ; 

Printer:  . . . ; 

end; 

2. 6. 2  Experiments 

Weinberg  is  probably  the  leading  proponent  of  the  use  of 
experiments  to  investigate  the  programming  process.  He  also 
suggests  that  the  results  of  experimental  psychology  be 
examined  to  determine  potential  areas  of  trouble  in  the 
programming  process  [Weinberg  1971].  For  example,  the 
principle  of  uniformity  indicates  that  it  is  easier  to 
remember  a  single  general  rule  than  a  set  of  specific  cases. 
To  illustrate  the  desirability  of  rhis  principle  in  the 
design  of  programming  languages,  Weinberg  asked  two  groups 
of  programmers  if  the  subscript  expression  21  -  K  was  valid 
in  their  respective  languages.  The  majority  of  the  FORTRAN 
programmers  questioned,  who  must  grapple  with  seven  specific 
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forms  of  legal  subscript  expressions,  were  unable  to  answer 
with  any  certainty.  The  PL/I  programmers  in  the  sample  were 
able  to  correctly  answer  that  this  expression  (and  "almost 
any"  other  expression)  was  a  valid  subscript  in  their 
language. 

An  experiment  was  performed  to  examine  non-syntactic 
errors  in  FORTRAN  programs  [Gould  and  Drongowski  1972], 
Thirty  subjects  were  asked  to  find  the  single  .semantic  or 
logical  error  in  one-page  listings.  The  subjects  were 
either  consultants  for  private  industry  or  a  university 
computing  center,  or  undergraduate  computer  science  students 
who  had  taken  at  least  three  courses.  The  listings  were 
chosen  from  an  IBM  statistical  package  and  the  distribution 
of  their  statements  approximated  the  findings  of  Knuth 
[ Knuth  1971].  It  was  felt  that  the  results  from  one-page 
listings  would  generalize  to  longer  programs  because  an 
error  is  often  quickly  isolated  to  within  forty  or  so 
statements.  Three  classes  of  errors  were  studied:  array, 
iteration,  and  assignment.  However,  no  effort  was  made  to 
seed  the  programs  with  errors  that  actually  occurred  in  the 
development  of  the  programs.  The  subjects  were  divided  into 
five  groups  based  on  the  debugging  aids  they  received.  All 
groups  received  listings  of  the  programs.  No  other  aids 
were  offered  to  the  control  group.  Another  group  was  given 
the  input  and  output  of  the  program,  while  a  third  group  had 
these  aids  plus  the  correct  output  for  the  program.  The 
fourth  group  was  told  the  class  of  error  the  problem 
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contained,  and  the  final  group  was  supplied  wixh  the  line 
number  containing  the  error. 

Subjects  were  timed  with  a  stopwatch.  When  a  subject 
believed  he  had  found  the  error  in  his  lisring,  the  watch 
was  stopped  and  he  described  the  suspected  error  to  the 
experimenter.  If  the  subject  was  incorrect,  the  watch  was 
restarted  and  he  began  debugging  again. 

Summing  over  all  groups,  67%  of  the  errors  were 
discovered  within  10  minutes  and  88%  of  the  errors  were 
discovered  within  20  minutes.  The  subjects  said  they  did 
not  debug  this  fast  in  actual  practice.  The  fast  debugging 
time  was  partly  due  to  the  high  motivation  of  the  subjects, 
who  felt  under  great  pressure  to  perform  well.  The  normal 
mode  of  debugging  for  these  subjects  was  to  execute  a 
program  containing  an  error  using  an  interactive  system. 
Curiously,  none  of  the  subjects  suggested  that  this  mode  of 
debugging  (with  strong  self-imposed  pressure  and  few  aids) 
could  be  more  efficient,  but  more  painful  than  their  usual 
practice.  Debugging  times  for  array  and  iteration  errors 
were  about  the  same,  but  debugging  times  for  assignment 
errors  were  much  longer.  While  the  array  and  iteration 
errors  violated  the  semantics  of  the  language,  the 
assignment  errors  in  this  study  did  not  and  thus  required 
knowledge  of  the  ’’deep  structure”  of  the  program  to  be 
identified.  With  the  exception  of  the  final  group,  the 
control  group  debugged  as  well  as  any  of  the  other  groups. 
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One  might  ^conclude  that  the  debugging  aids  provided  were  of 
no  help.  The  final  group,  which  was  told  the  number  of  the 
line  containing  the  error,  debugged  twice  as  efficiently  as 
the  other  groups.  However,  the  experimenters  suggested  that 
since  telling  the  programmer  the  number  of  the  line  in  error 
is  perhaps  the  ultimate  debugging  aid,  the  improvement 
measured  may  be  an  upper  bound  on  the  degree  of  improvement 
that  debugging  aids  could  provide  for  this  type  of  error. 

These  results  were  replicated  in  a  later  experiment 
[Gould  1973].  Ten  experienced  FORTEAN  programmers  at  an  IBM 
research  center  debugged  twelve  FORTRAN  listings,  each  of 
which  contained  a  single  non- syntactic  error.  Median 
debugging  time  was  seven  minutes,  the  errors  were  not  found 
in  11%  of  the  listings,  and  less  than  one  incorrect 
identification  of  the  error  was  made  on  each  listing. 
Assignment  errors  were  about  three  times  as  difficult  to 
detect  as  array  or  iteration  errors. 

Weissman  attempted  to  measure  how  well  programmers 
understand  programs  [Weissman  1974].  He  constructed  a  set 
of  programs  in  which  various  factors  that  may  affect  the 
"psychological  complexity"  of  a  program  (e.g.,  mnemonic 
variable  names)  occurred  at  various  levels  (e.g.,  long, 
meaningful  names  or  single-character  identifiers) .  The 
subjects  were  asked  to  read,  study,  and  modify  one  of  the 
programs.  Subjective  and  objective  measurements  of 
understanding  were  taken  throughout  the  experiment.  The 
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subjective  measure  of  understanding  was  a  programmer's  self- 
evaluation  of  how  well  he  felt  he  understood  the  program. 
Various  objective  measurement  techniques  included  a  hand- 
simulation  of  the  values  of  variables,  a  quiz  about  the 
program,  and  a  subjective  judgement  of  the  quality  of  the 
modifications  performed  by  the  programmer.  Statistically 
significant  results  were  obtained  for  several  factors  that 
affect  program  readability:  mnemonic  variable  names, 
comments,  and  paragraphing. 

In  an  experiment  described  by  Miller,  12  college 
subjects  without  previous  programming  experience  were 
trained  for  55  minutes  and  asked  to  solve  six  card-sorting 
problems  using  an  interpretive,  English-like  laboratory 
programming  language  [Miller  1974].  The  specifications  of 

the  problems  differed  in  the  manner  in  which  the  required 
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tests  were  expressed  (i.e.,  affirmatively  or  negatively)  and 
the  manner  in  which  the  logical  relations  to  be  satisfied  by 
the  outcome  of  the  tests  were  stated  (i.e.,  conjunction  or 
inclusive  disjunction) .  The  amount  of  time  spent 
programming  and  the  number  of  errors  that  occurred  indicated 
that  problems  containing  negatively- expressed  tests  were 
more  difficult  to  solve  than  problems  containing 
affirmatively-expressed  tests.  Also,  problems  containing 
disjunctive  relations  were  more  difficult  to  solve  rhan 
problems  containing  conjunctive  relations.  Subjects  spent 
more  time  and  made  more  program  modifications  in  debugging 
solutions  than  in  preparing  them  initially.  No  syntactic 
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errors  were  tabulated  in  this  study  because  the  commands 
were  selected  by  letter,  rather  than  generated  by  the 
subjects.  Illegal  addresses  in  the  commands  were  detected 
immediately  and  prevented  by  the  interpreter. 

Rather  than  comparing  entire  languages,  Sime,  Green,  and 
Guest  advocated  devising  micro-languages  stressing  specific 
features  common  to  many  languages  [Sime  et  al.  1971].  They 
designed  two  languages,  each  of  which  contained  one  of  two 
conditional  constructs:  a  nestable  two-tailed  consrruction 
(with  no  conjunction  or  disjunction  operations)  similar  to 
that  of  ALGOL  60,  and  a  branch-t c-label  construction  (with 
forward  labels  only)  used  in  many  low-level  languages.  To 
compare  these  constructs,  18  subjects  without  previous 
programming  experience,  half  working  in  each  language,  were 
each  asked  to  solve  five  card  sorting  problems.  Using  an 
interactive  system,  subjects  constructed  programs  by  picking 
words  from  a  dictionary.  When  a  subject  was  satisfied  that 
his  program  was  complete,  he  notified  the  system  which  then 
checked  the  program  for  syntactic  errors.  Syntactically 
correct  programs  were  run  by  the  system.  Otherwise  the 
source  was  returned  to  the  subject  for  modification.  The 
use  of  simple  problems  allowed  the  system  to  check  that  the 
solutions  were  correct.  Programs  containing  semantic  errors 
were  returned  to  the  subject  for  modification.  While  all 
nine  subjects  using  the  nestable  conditional  construct 
completed  all  five  . problems  in  90  minutes,  only  five 
subjects  using  the  branch-to-label  construct  did  so. 
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Statistical  comparisons  were  made  using  data  from  only  those 
subjects  who  completed  all  problems.  Statistically 
significant  results  were  obtained  demonstrating  that  the 
subjects  using  the  nestable  conditional  construct  made  fewer 
semantic  errors  and  spent  less  time  arriving  at  solutions 
than  did  subjects  using  the  branch-ro- label  construct.  The 
former  group  had  greater  difficulty  solving  the  more  complex 
problems,  probably  because  cf  increased  levels  of  nesting. 
This  suggests  that  syntactic  devices  (e.g.,  conjunction  and 
disjunction  operations)  be  used  to  reduce  the  nesting  depth. 

2 . 7  R e lat i on  to  previous  work 

The  few  studies  that  have  been  done  thus  far  have  not 
produced  the  empirical  evidence  needed  to  verify  that 
particular  constructs  enhance  the  reliability  of  software. 
The  observations  of  Moulton  and  Muller,  Nagy  and  Pennebaker, 
and  Boies  and  Gould  were  too  crude  to  yield  the  necessary 
evidence.  The  experiments  of  Gould  and  Drongowski  and  those 
of  Weissman  did  not  deal  with  program  construction. 
Subjects  without  previous  programming  experience,  using 
laboratory  programming  languages,  were  studied  by  Miller  and 
by  Sime,  Green,  and  Guest.  Additionally,  Miller  dealt  only 
with  the  selection  and  organization  of  commands,  not  their 
generation . 

This  thesis  draws  most  heavily  on  the  work  of  Ichbiah 
and  Rissen,  and  of  Youngs.  An  existing  language  was 
examined  for  characteristic  errors,  and  an  attempt  was  made 
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to  design  them  out  of  the  language.  Many  of  the  techniques 
of  Youngs  (e.g.,  his  coding  scheme)  were  adapted  for  this 
study.  The  major  failing  of  Youngs*  observation  was  its 
lack  of  conclusions  about  language  design  because  of  its  use 
of  several  languages  with  widely  differing  features,  making 
comparisons  between  individual  language  features  difficult. 
By  controlling  the  differences  in  two  languages  and  using  a 
detailed  error  classification  scheme,  this  study  was  able  to 
gather  the  appropriate  data  with  which  to  compare  language 
features . 

Some  of  the  language  features  described  in  this  chapter 
are  common  to  both  of  the  languages  used  in  this  thesis: 
disciplined  control  structures,  type  checking,  and  mandatory 
declaration  of  data  items.  However,  additional  desirable 
features  were  added  to  one  of  the  languages  in  an  attempt  to 
enhance  the  reliability  of  software  produced  using  the 
language;  constant  identifiers,  reserved  words  and  spelling 
repair,  redundant  control  structures,  explicit  interfaces, 
and  other  features  intended  to  exploit  the  backgrounds  of 
the  subjects. 
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CHJ^-PTER  THREE 


METHODOLOGY 


3 . 1  Overview 

The  starting  point  for  this  work  was  a  small  existing 
language  with  constructs  that  support  concurrency,  TOPPS 
[Czarnik  et  al.  1973],  Using  some  of  the  techniques 
outlined  in  Chapter  Two,  and  examining  the  language  for 
characteristic  errors,  a  number  of  mutually  independent 
language  features  were  redesigned  in  an  attempt  to  reduce 
the  number  and/or  the  severity  of  the  errors  made  in 
programming  with  the  language.  TOPPSII  [Gannon  1973] 
contained  the  redesigned  features,  but  was  otherwise 
identical  to  TOPPS.  Descriptions  of  the  two  languages  can 
be  found  in  Chapter  Four.  A  new  compiler  was  constructed 
and  a  new  manual  prepared.  Student  programmers  were  divided 
into  two  groups,  one  working  in  each  language,  to  obtain 
empirical  evidence  about  the  effect  of  the  twc  languages  on 
the  programs  written  by  the  students. 

The  empirical  evidence  presented  in  this  thesis  is 
derived  from  the  errors  made  by  students  programming 
solutions  to  two  problems.  For  the  purposes  of  this  study, 
TOPPSII  was  judged  to  enhance  the  reliability  of  software 
more  than  TOPPS  if  the  errors  committed  by  the  users  of 
TOPPSII  were  less  frequent  and/or  less  severe  than  the 
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errors  committed  by  the  users  of  TOPPS.  In  addition  to  this 
main  hypothesis,  hypotheses  were  formed  concerning  the 
frequency  and  severity  of  the  errors  that  were  attributable 
to  each  language  feature  that  was  redesigned. 

This  chapter  contains  an  explanation  of  the  limitations 
placed  on  this  study  and  a  discussion  of  the  various  steps 
of  the  methodology.  Section  3.3  outlines  the  techniques 
used  to  redesign  TOPPS  and  to  observe  the  programming 
process.  Section  3.4  discusses  the  assignment  of  subjects 
to  the  two  languages,  and  the  backgrounds  of  the  resulting 
groups  of  subjects.  Next,  the  task  and  directions  given  to 
the  subjects  are  described.  Section  3.6  explains  the 
identification  and  classification  of  errors.  Section  3.7 
contains  definitions  of  the  measures  used  to  compare  errors. 
Finally,  the  statistical  techniques  used  to  evaluate  the 
data  are  discussed. 

3 . 2  Initial  constraints 

Three  initial  constraints  were  placed  on  this  study: 
student  subjects  would  be  used,  the  language  features  being 
studied  would  be  easy  to  implement,  and  the  data  collection 
mechanism  would  be  inexpensive  to  operate.  The  students 
selected  were  part  of  a  fourth  year  course  in  operating 
systems  offered  at  the  University  of  Toronto  in  the  spring 
of  1974  by  Professor  D.C.  Tsichritzis.  This  course  was 
selected  for  several  reasons.  Foremost  among  them  was  its 
use  of  the  programming  language  TOPPS.  This  language  and 
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its  compiler  are  small  and  easily  modified.  Also,  like  many 
student  languages,  TOPPS  is  not  widely  used  and  therefore  it 
was  safe  to  assume  that  the  subjects  would  not  be  biased  by 
prior  familiarity  with  it.  Another  advantage  offered  by  the 
use  of  such  an  advanced  course  is  that  the  students  are 
relatively  experienced  programmers.  Weinberg  has  commented 
that  traditional  psychology  is  the  psychology  of  eighteen- 
year-old  college  freshmen.  That  is,  the  typical  source  of 
subjects  for  experiments  is  the  freshman  psychology  course 
in  a  university,  where  one  of  the  requirements  for  credit  is 
participation  as  a  subject  in  an  experiment.  Following  this 
example  in  experiments  on  the  programming  process  (i.e., 
using  students  in  a  first  year  programming  course)  would 
probably  result  in  an  overemphasis  of  trivial  errors 
associated  with  learning  programming. 

The  problems  selected  for  this  study  were  those  that  had 
been  used  in  the  course  in  previous  years.  Using  the  normal 
class  assignments  provides  problems  that  are  well-tried  and 
that  student  programmers  are  well-motivated  to  complete. 

A  second  constraint  was  that  the  experimenter  avoid 
becoming  involved  in  a  large  implementation  project.  In 
general,  only  constructs  that  cculd  be  implemented  by 
altering  the  TOPPS  compiler  without  changing  the  TOPPS 
interpreter  were  redesigned.  (The  TOPPS  processor  consists 
of  a  compiler  that  produces  pseudo-code  for  a  TOPPS 
machine.) 
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A  final  constraint  was  that  the  mechanisms  used  for 
observing  the  programming  process  not  be  expensive  to  use. 
As  each  submission  to  the  language  processor  by  each  student 
was  to  be  preserved,  the  mechanism  had  to  be  kept  simple. 

3 . 3  Preparation  for  ^e  study 

In  order  to  perform  the  study  it  was  necessary  to  choose 
the  language  features  to  be  altered,  implement  them,  and 
produce  documentation  describing  the  new  language. 
Additionally,  a  mechanism  had  to  be  implemented  to  observe 
submissions  to  the  language  processor.  The  study  was 
performed  on  an  IFM  System/370  165  in  a  batch  processing 
environment. 

The  language  design  decisions  were  made  in  the  spring  of 
1973.  During  that  summer,  the  TOPPSII  processor  was 
implemented  and  a  users*  manual  for  the  language  produced. 
In  late  September,  the  language  was  used  in  an  operating 
systems  class  to  discover  errors  in  the  implementation.  The 
observation  mechanism  was  implemented  during  the  fall  of 
1973  and  was  first  used  in  late  November.  The  experiment 
described  in  this  thesis  was  performed  during  January  and 
February  of  1974. 

3.3.1  Preparing  the  language 

Two  methods  were  used  to  select  language  features  to  be 
altered.  First,  informal  observations  of  the  programming 
process  were  made  in  two  courses.  The  first  course  was  a 


45 


graduate  course  in  computer  program  engineering  offered  in 
the  fall  of  1972  in  which  students  wrote  programs  in  PL/I, 
ALGOL  W,  and  the  Project  SUE  System  Language,  In  the  spring 
of  1973,  observations  were  made  in  the  same  operating 
systems  course  used  in  the  final  study,  in  which  students 
programmed  in  TOPPS.  Students  in  these  courses  were  asked 
to  submit  as  many  intermediate  listings  as  possible  and  to 
summarize  the  errors  on  each  listing  according  to  a  crude 
classification  scheme.  The  experimenter  examined  the 
listings  to  get  a  feeling  for  the  type  and  number  of  errors 
committed.  No  attempt  was  made  either  to  obtain  complete 
sets  of  intermediate  listings  or  to  statistically  analyze 
the  results  of  the  informal  observations.  The  second  method 
used  to  select  language  features  to  be  altered  was  an 
introspective  analysis  of  the  characteristic  errors  of 
TOPPS.  Using  the  criteria  suggested  by  Ichbiah  and  Riessen 
in  Section  2.6.1,  the  experimenter  examined  the  constructs 
of  TOPPS  for  characteristic  errors. 

Eventually  it  was  decided  that  the  redesign  should 
encompass  the  following  nine  language  features: 

•  order  of  expression  evaluation 

•  assignment  symbol 

•  relation-connectors 

•  semicolon  as  statement  punctuation 

•  selection  statements 

•  iteration  statements 

•  delimiters  for  compound  statements 

•  inheritance  of  environment 

•  constants 

A  more  complete  discussion  of  the  differences  in  the 
features  of  the  two  languages  and  a  discussion  of  rhe  errors 
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attributable  to  each  feature  appears  in  Chapter  Four,  With 
the  exception  of  additional  selection  and  iteration 
statements,  all  the  changes  were  effected  without  making 
alterations  to  the  interpreter  for  TOPPS. 

The  documentation  for  TOPPS  was  a  users*  manual  for  the 
language  and  four  sample  programs.  An  attempt  was  made  to 
produce  equivalent  documentation  for  TOPPSII.  Using  an 
interactive  text  editor,  the  users*  manual  was  altered  to 
describe  the  features  of  the  new  language.  Also,  the  four 
sample  programs  were  rewritten  and  executed  using  the 
TOPPSII  language  processor. 

3.3.2  The  data  collection  mechanism 

As  a  safeguard  against  uncooperative  subjects,  a  data 
collection  mechanism  (drain)  was  used  to  copy  each  program 
submitted  by  each  subject.  In  order  that  the  draining 
procedure  be  inexpensive,  it  was  decided  to  copy  the  input 
cards  of  the  subjects,  rather  than  the  output  of  the 
language  processors.  This  was  done  principally  to  avoid 
copying  the  output  of  jobs  that  entered  infinite  loops 
containing  output  statements.  However,  draining  cards  had 
another  potential  advantage,  as  it  would  have  allowed 
programs  to  be  rerun  at  a  later  date  with  a  different 
language  processor  containing  more  statistics-gathering 
routines . 
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The  language  processors  for  both  TOPPS  and  TOPPSII  each 
had  their  respective  drains  builr  into  the  catalogued 
procedures  for  accessing  the  processors.  The  drain  was 
activated  by  the  first  card  of  a  subject's  source  code. 
This  card  had  to  be  a  comment  card  containing  the  following 
four  parameters:  the  name  of  the  subject,  the  name  of  the 
problem  the  subject  was  attempting  to  solve,  the  number  of 
the  submission  for  that  problem,  and  the  time  (in  minutes) 
that  the  subject  spent  preparing  the  run  (excluding 
keypunching) .  If  this  information  was  acceptable,  the 
subject's  input  source  code  and  input  data  were  copied  to  a 
disk  file.  When  this  file  became  full,  it  was  automatically 
dumped  to  tape  and  emptied.  A  more  complete  description  of 
the  drain  appears  in  Appendix  A. 

The  obvious  appeal  of  this  scheme  is  its  automation.  No 
intervention  by  the  experimenter  was  required  when  disk 
files  became  full.  The  drain  also  met  the  requirement  that 
it  not  be  too  expensive,  increasing  the  cost  of  $.50  jobs  by 
about  25%, 

3 . 4  The  subjects 

Initially,  51  subjects  from  the  operating  systems  course 
were  to  participate  in  the  study.  The  course  is  a  fourth- 
year  course,  open  to  both  fourth-year  students  and  graduate 
students.  The  course  was  offered  as  part  of  the  extension 
program  so  that  both  part-time  and  full-time  students  were 
registered.  It  was  felt  that  these  factors  might  lead  to 
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groups  with  nonhomogGreous  ability.  In  particular,  graduate 
students  might  be  expected  to  have  had  more  experience  than 
undergraduates,  and  full-time  students  might  be  expected  to 
have  more  time  to  work  on  problems  than  part-time  students. 
With  this  in  mind,  the  students  were  partitioned  in  four 
groups  based  on  these  factors.  Within  the  groups,  students 
were  listed  alphabetically  and  alternately  assigned  to  work 
in  one  of  the  two  languages.  In  addition,  a  questionnaire 
was  later  distributed  to  determine  the  programming 
experience  of  the  subjects. 

Ten  of  the  25  subjects  who  started  in  TOPPS  and  15  of 
the  26  subjects  who  started  in  TOPPSII  finished  both 
problems.  As  was  the  case  in  the  study  by  Sime,  Green,  and 
Guest  [Sime  et  al.  1973],  only  the  data  of  those  subjects 
completing  solutions  to  both  problems  was  considered  in  the 
analysis  of  the  results.  Assuming  that  the  two  groups 
initially  contained  subjects  of  similar  abilities,  any  bias 
introduced  by  comparing  only  the  subjects  who  finished  both 
problems  would  be  in  the  negative  direction.  That  is, 
differences  favoring  the  TOPPSII  group  would  be  achieved 
despite  comparing  the  "best”  15  TOPPSII  subjects  to  the 
"best"  10  TOPPS  subjects. 

The  questionnaires  submitted  by  these  subjects  are 
summarized  in  the  tables  on  the  following  pages.  Both 
groups  of  subjects  contained  approximately  equal  proportions 
of  part-time  to  full-time  students  (six  of  ten  in  TOPPS  and 
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seven  of  fifteen  in  TOPPSII)  and  of  graduates  to 
undergraduates  (seven  of  ten  in  TOPPS  and  eleven  of  fifteen 
in  TOPPSII) .  The  mean  course  examination  score  of  the 
subjects  working  in  TOPPS  was  slightly  higher  than  the  mean 
score  of  subjects  working  in  TOPPSII  (69.50  against  66.80). 
However,  this  difference  was  not  statistically  significant 
using  the  Wilcoxon-Mann-Whitney  test  described  in  Section 
3.8.1.  Half  of  the  TOPPS  subjects  had  more  than  three  years 
professional  experience  while  only  one-fifth  of  the  TOPPSII 
subjects  had  similar  experience.  The  difference  in  the 
years  of  professional  experience  was  mildly  significant:  (at. 
the  <20%  level) .  Large  numbers  of  subjects  in  each  group 
listed  FORTPAN  among  their  most  frequently  used  languages 
(seven  of  ten  TOPPS  subjects  and  ten  of  fifteen  TOPPSII 
subjects) . 
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TOPPSII  Subjects 


3  •  5  Assign  ins  task 


The  experimenter  met  with  the  class  to  explain  the 
purpose  of  the  study.  At  this  meeting,  users*  manuals  for 
the  languages,  each  tagged  with  a  subject’s  name,  were 
distributed.  The  subjects  were  expected  to  learn  the 
language  from  the  manual.  To  reduce  bias  introduced  by  the 
experimenter,  no  tutorials  explaining  the  languages  were 
given.  Instead,  one  week  after  the  subjects  received  their 
manuals,  a  question  and  answer  session  on  the  languages  was 
held.  Answers  were  provided  by  the  experimenter,  who 
labelled  them  according  to  the  language  to  which  they 
applied.  All  further  questions  posed  by  the  subjects  were 
answered  by  the  experimenter  on  an  individual  basis. 

The  subjects  were  asked  to  program  twc  problems  whose 
solutions  totaled  123  to  306  statements.  (These  figures 
appear  in  Appendix  H.2.)  The  first  problem  involved 
simulating  the  card  game  "spoons”  with  five  players  and  four 
spoons.  The  five  players  are  each  dealt  four  cards  and  pass 
cards  in  a  circle  until  one  player  collects  four  of  the  same 
kind.  At  this  point,  the  player  yells  "geronirao”  and  all 
the  players  grab  for  spoons.  The  loser  is  the  player  who 
does  not  receive  a  spoon.  The  second  problem  dealt  with  a 
spooling  system  containing  an  input  spooler,  an  output 
spooler,  and  two  central  processing  units.  The  input 
spooler  reads  in  jobs  consisting  of  a  character  string  and 
its  replication  factor.  The  jobs  are  then  passed  to  a 
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central  processing  unit  that  reverses  the  character  string 
and  produces  the  number  of  copies  specified  by  the 
replication  factor.  The  completed  jobs  are  printed  by  the 
output  spooler.  Full  descriptions  and  solutions  for  the 
problems  can  be  found  in  J^.ppendix  B. 

In  order  to  reduce  the  cost  of  the  study,  each  subject 
was  asked  to  submit  his  intermediate  listings  to  the 
experimenter.  Each  subject  was  also  asked  to  complete  a 
questionnaire  that  was  printed  as  the  first  page  of  his 
output.  To  avoid  alerting  them  to  types  of  potential 
errors,  subjects  were  requested  to  provide  only  the  number 
of  the  source  statement  containing  the  error  and  a  general 
English  language  description  of  the  cause  of  the  error. 
Finally,  each  subject  was  told  to  place  a  comment  card  that 
activated  the  drain  as  the  first  card  in  his  source  program. 

3 . 6  Tabulating  the  err or s 

The  results  were  evaluated  in  two  steps.  First,  the 
errors  in  each  run  were  identified  and  classified  using  an 
error  taxonomy  similar  to,  but  more  detailed  than,  that  of 
Youngs.  Next,  problem  submission  information  (e.g., 
programmer  and  problem  identification)  and  the  errors  that 
occurred  in  the  submission  were  encoded  and  entered  into  a 
sequential  file.  The  error  taxonomy  appears  in  Appendix  C, 
and  the  format  of  each  of  the  records  used  to  describe  a 
submission  appear  in  Appendix  D. 
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3.6.1  Identifying  the  errors 


The  listings  of  the  subjects  who  completed  both  problems 
were  examined  by  the  experimenter  fcr  errors.  A  detected 
error  was  indicated  by  the  appearance  of  a  diagnostic 
message  or  incorrect  problem  output.  (Obviously,  it  is  a 
matter  of  opinion  whether  or  not  incorrect  problem  output  is 
sufficient  for  a  subject  to  detect  an  error.)  Undetected 
errors  fall  into  two  categories:  errors  marked  in  the 
listing  than  the  programmer  discovered  by  desk-checking 
without  the  help  of  either  diagnostic  messages  or  incorrect 
program  output,  and  errors  detected  by  the  experimenter 
during  his  examination  of  (submitted)  solutions  to  the 
problems. 

In  runs  following  the  run  on  which  the  error  appeared,  a 
subject  may  have  changed  his  program,  either  correcting  the 
error  or  altering  the  manner  in  which  the  error  manifested 
itself.  Whether  or  not  a  diagnostic  message  or  incorrect 
problem  output  was  produced  on  subsequent  runs,  the  error 
was  said  to  persist  if  it  remained  uncorrected. 

To  determine  the  number  of  occurrences  of  the  errors, 
both  detected  and  undetected  errors  were  traced  back  to 
their  origin,  and  counted  on  all  intervening  runs.  Errors 
that  occurred  in  the  source  code  were  traced  back  to  the  run 
in  which  the  compiler  first  analyzed  the  source  code. 
Errors  that  occurred  in  the  input  data  were  traced  back  to 
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the  run  in  which  the  program  first  reached  execution  of  an 
input  statement. 

3.6.2  Cl  as  sif  ;Ying  submissions 

Following  conventions  established  by  Youngs,  each 
submission  was  classified  to  identify  the  subject  and  the 
problem  he  was  attempting  to  solve,  the  number  of  each  type 
of  statement  used  in  the  submission,  and  the  errors  that 
were  observed.  Except  for  the  first  column  of  each  record, 
which  is  used  to  differentiate  the  three  types  of  records  in 
the  file  (programmer  identification,  statement  profile,  and 
error  description) ,  an  attempt  was  made  to  keep  the  data  on 
a  single  record  in  one  format  (i.e.,  either  all  numeric  or 
all  character)  to  avoid  keypunching  mistakes. 

Information  identifying  a  subject  and  the  problem  he  was 
trying  to  solve  appears  on  a  programmer  identification 
record.  This  record  also  indicates  the  number  of  error 
description  records  associated  with  this  submission. 

The  second  record  for  each  submission  contains  the 
number  of  each  type  of  statement  used  in  the  program.  These 
figures  are  computed  by  the  compilers  for  the  two  languages. 
Unfortunately,  when  a  particularly  severe  error  is 
encountered  or  the  limit  on  the  number  of  syntax  errors 
allowed  on  a  run  is  exceeded,  the  remaining  source  lines  are 
skipped  and  the  figures  are  unreliable.  Also,  attempts  to 
compute  the  number  of  expressions,  assignments,  statements. 
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and  parenthesized  blocks  in  TOPPS  are,  at  best,  only 
approximations  based  on  the  number  of  times  their  respective 
productions  in  the  TOPPS  grammar  are  applied. 

Error  description  records  contain  information  about  the 
cause  and  location  of  each  error,  as  well  as  the  location  of 
the  best  detection  information  for  the  error.  The  errors 
are  coded  in  terms  of  their  cause,  not  in  terms  of  their 
eventual  manifestation.  An  example  will  now  be  given  to 
illustrate  this  difference.  The  subscript  of  the  array  Arr 
is  out  of  bounds  during  the  last  repetition  of  the  loop 
because  the  termination  condition  of  the  repeat  construct 
does  not  agree  with  the  declared  bounds  of  the  array. 

variable  I,X; 

array  Arr  bound  4 ; 

• 

• 

I  :=  0; 

repeat ; 

• 

• 

I  :=  I  +  1; 

X  :=  Arr  (I) ; 

until  I  >  4 ; 

This  error  would  be  regarded  as  due  to  an  incorrectly 
specified  termination  condition  on  a  repeat  statement  rather 
than  to  the  value  of  the  subscript  being  out  of  the  declared 
range.  This  example  illustrates  one  of  the  problems  of  the 
coding  scheme,  the  amount  of  judgement  reguired  by  the 
experimenter  in  ceding  errors. 

The  general  causes  of  errors  were  divided  into  clerical, 
syntactic,  semantic,  logical,  influence  of  the  other 
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language,  and  avalanche  (i.e.,  spurious  diagnostics 
generated  by  the  parser  during  the  application  of  recovery 
techniques) .  The  specific  causes  of  errors  enable  much 
finer  distinctions  to  be  made.  In  the  example  above,  the 
specific  cause  would  be  "termination  condition  incorrectly 
specified  due  to  causes  other  than  zero  indexing  on  array 
elements  or  character  positions.”  The  phrase  categorization 
is  an  attempt  to  attribute  the  error  to  a  particular  symbol. 
The  location  of  the  best  error  detection  information  is 
either  the  scanner,  parser,  synthesize  routines, 
interpreter,  special  diagnostic  instructions,  system 
messages,  problem  output,  dump,  or  deadlock  analysis.  The 
values  that  the  various  fields  are  allowed  to  contain  are 
described  in  a  taxonomy  in  Appendix  C.  The  characters 
preceding  the  appropriate  description  in  the  taxonomy  are 
entered  in  the  appropriate  fields  of  an  error  description 
record.  In  the  example  above,  if  the  error  were  detected  by 
the  interpreter,  the  error  record  would  be  as  follows: 

2  C  FCG  DEAC  CGA  BA 

The  taxonomy  is  not  without  its  problems.  For  example, 
if  two  symbols  are  interchanged,  the  phrase  containing  the 
error  is  difficult  to  specify.  The  convention  adopted  was 
that  the  phrase  in  error  is  the  first  symbol.  Many  problems 
of  a  preliminary  coding  scheme  were  discovered  during  a 
trial  with  listings  from  the  operating  systems  course 
offered  in  the  fall  of  1973. 


58 


3.7  Measures 


The  classified  errors  were  examined  to  determine  the 
number  of  errors  and  occurrences  of  errors  as  described  in 
Section  3.6.1.  Also,  two  measures  of  severity  were 
calculated:  the  persistence  of  errors  and  the  number  of 
runs  containing  errors  (hereafter  referred  to  as  error 
runs) .  Unfortunately,  most  subjects  did  not  change  their 
indicated  preparation  time  from  run  to  run,  so  plans  to  use 
this  measure  were  abandoned. 

The  persistence  of  an  error  is  a  measure  of  the  number 
of  runs  in  which  an  error  occurred  (i.e.,  from  the  origin  of 
the  error  until  it  was  either  eliminated,  by  correction  or 
removal  from  the  program,  or  the  solution  to  the  problem  was 
submitted) .  It  is  calculated  by  dividing  the  number  of 
occurrences  of  errors  by  the  number  of  errors.  The 
existence  of  errors  in  final  solutions  to  problems  means 
that  persistence  is  only  a  lower  bound  on  the  severity  of 
these  errors  (i.e.,  they  would  exist  for  at  least  one  more 
run)  . 

Finally,  the  number  of  runs  that  contained  an^  of  the 
errors  that  are  part  of  a  particular  hypothesis  is 
calculated  (i.e.,  the  number  of  runs  ruined  because  of  a 
particular  language  feature)  .  Like  persistence,  error  runs 
is  an  attempt  to  measure  the  severity  of  errors,  rather  than 
just  the  gross  number  of  errors.  An  error  that  persists  for 
five  runs  will  have  error  runs  equal  to  five,  while  five 
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similar  errors  that  occur  on  the  same  run  will  have  error 


runs  equal  to  one. 

3 . 8  Statistical  t echn iques 

The  averages  (per  programmer  for  the  number  of  errors, 
occurrences,  and  error  runs,  and  per  error  for  persistence) 
were  calculated  for  each  language  for  each  hypothesis. 
Because  averages  over  small  samples  can  be  greatly  distorted 
by  one  or  two  very  bad  performances,  a  non- parametric, 
small-sample  test  (the  Wilcoxon- Kann-Whitney  test  [Freund 
1967])  was  used  to  determine  the  statistical  significance  of 
all  of  the  differences  between  errors,  occurrences,  and 
runs,  as  well  as  the  difference  between  persistences  for 
each  hypothesis.  In  the  cases  where  a  large  number  (i.e., 
greater  than  100)  of  errors  were  available,  a  large-sample 
comparison  of  means  [Lindgren  1966]  was  used  to  compare 
persistences . 

3.8.1  Wilcoxon-Mann-Whitney  test 

The  Wilcoxon  test  was  selected  because  it  is  not  related 
specifically  to  the  parameters  of  a  given  population,  and 
may  therefore  be  used  under  very  general  conditions.  In 
contrast,  a  standard  parametric  test  (e.g.,  the  t-test)  may 
be  inappropriate  if  one  of  the  sample  populations  displays 
considerably  pore  variance  than  another.  This  was  the  case 
since  both  three  of  the  five  best  programming  performances, 
and  the  worst  four  programming  performances  were  observed 
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among  the  TOPPS  subjects.  Attempts  to  reduce  the  variance 
by  attributing  it  to  the  varying  ability  of  the  subjects  as 
measured  by  their  examination  grades  failed  because  there 
was  no  significant  relation  between  the  grades  and  either 
the  number  of  errors  or  the  number  of  occurrences  of  errors. 

The  Wilcoxon  test  is  not  without  its  disadvantages.  The 
test  is  wasteful  of  information  because  it  compares  the 
ranks  of  the  values  of  two  populations,  rather  than  the 
values  themselves.  This  leads  to  a  greater  risk  of 
accepting  a  false  null  hypothesis  than  the  alternative 
standard  tests.  That  is,  the  Wilcoxon  test  requires  a 
pronounced  difference  from  the  null  hypothesis  to  indicate 
statistical  significance. 

The  null  hypothesis  is  that  both  samples  come  from 
identical  populations.  This  implies  that  the  means  of  the 
ranks  assigned  to  the  values  of  the  two  sample  populations 
should  be  approximately  the  same.  In  this  study  the 
alternative  hypothesis  was  that  the  TOPPSII  population  would 
have  smaller  mean  numbers  of  errors,  occurrences, 
persistence,  and  runs.  More  of  the  smaller  ranks  would  then 
have  gone  to  values  in  the  TOPPSII  sample  and  the  larger 
ranks  to  values  in  the  TOPPS  sample. 

To  perform  the  Wilcoxon  test  on  sample  populations  of 
sizes  m  and  n,  the  values  of  the  two  sample  populations  are 
ranked  jointly,  as  if  they  were  one  sample,  in  increasing 
order  of  magnitude.  The  values  of  the  joint  population  are 
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then  assigned  the  ranks  1 , 2, . . . , m+n.  For  equal  values,  each 
value  is  assigned  the  mean  of  the  ranks  that  the  values 
jointly  occupy.  (Z.g.,  if  the  fourth  and  fifth  values  are 
identical,  they  would  each  be  assigned  the  rank 
(4-t-5) /2  =  4.5.)  The  sampling  distribution  0  is  computed  by 

0  =  m  *  n  +  (m’*' (m+ 1) /2)  - 

where  is  the  sum  of  the  m  ranks  of  the  first  sample. 
Under  the  null  hypothesis,  that  the  m+n  values  come  from  a 
single  population,  U  has  mean; 

Mu(U)  =  m  *  n  /  2 
and  standard  deviation: 

Sigma  (U)  =  square  root(m  *  n  *  (m-^n  +  l)  /  12) 

If  m  and  n  are  both  greater  than  eight,  U  can  be 
approximated  closely  with  a  normal  distribution.  The  null 
hypothesis  can  then  be  rejected  if  the  value  of  a  standard 
normal  variable  Z  is  less  than  the  level  of  significance  of 
the  test.  The  value  of  Z  is  computed  by 

Z  =  (U-Mu  (U)  )  /  Sigma  (U) 

The  assumption  of  continuous  sample  populations  must  be 
made  to  allow  the  possibility  of  ties  to  be  ignored  in  the 
derivation  of  Sigma (U).  A  correction  in  the  calculation  of 
Sigma  (U)  to  account  for  ties  can  be  made,  but  it  is 
generally  negligible  unless  the  number  of  ties  is  large. 

The  level  of  significance  at  which  the  test  is  being 
conducted  is  the  probability  of  committing  a  Type  I  error 
(i.e. ,  falsely  rejecting  the  null  hypothesis) .  For  the 
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alternative  hypotheses  Mu  (n)  >  Mu  (m)  (i.e,,  one-sided  test) 

and  Mu(n)  -•=  Mu(m)  (i.e.,  two-sided  test),  the  critical 
values  at  the  levels  of  significance  are  as  follows; 


Level 

One-sided 

value 

Two-sided  value 

<20% 

0.84 

1  .28 

<10% 

1.28 

1.64 

<5% 

1.64 

1.96 

<1% 

2.  33 

2.58 

<.5% 

2.58 

2.81 

<.  1% 

3.09 

3.29 

If  either  m 

or  n  is  smaller 

than  eight,  the  normal  curve 

approximation 

can  not  be 

used. 

Instead,  the  test  must  be 

based  on  statistical  tables,  e. 

g.  ,  [ Owen  1962]. 

3.8.2  Large-scale  comparison  of  means 

Given  two  independent  samples  of  sizes  m  and  n  with 
means  X  and  Y  and  variances  Sigma^  and  Tau2  respectively, 
the  difference  X  -  Y  is  asymptotically  normal,  for 
sufficiently  well-behaved  population  distributions.  If 
Sigma2  and  Tau^  are  known,  the  ‘Statistic  Z  is  approximately 
a  standard  normal  variable  under  the  hypothesis  of  equal 
means  when 

Z  =  (X  -  Y)  /  square  root CSigma^/m  +  Tau^/n) 

Population  variances  are  seldom  known.  However,  in  large 
samples,  the  sample  variances  (i.e.,  s2  and  t^)  are  usually 
close  to  the  population  variances  (for  even  better  behaved 
distributions).  Therefore,  Z  can  be  replaced  by  Z*  with 
reasonable  success  when 

Z*  =  (X  -  Y)  /  sguare^root (s2/m  *  t^/n) 
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CHAPTER  FOUR 


A  COMPARISON  OF  LANGUAGE  FEATURES 

4 . 1  Introduction 

This  chapter  contains  a  description  of  some  of  the 
features  of  TOPPS  and  their  characteristic  errors.  After 
this,  the  individual  decisions  made  in  designing  TOPPSII  are 
discussed.  Particular  attention  is  paid  to  the  errors 
attributable  to  the  features  of  TOPPS  whose  effect  was  to  be 
tempered  in  TOPPSII,  An  evaluation  of  the  expected  impact 
of  each  feature  on  reliability  is  offered.  More  detailed 
information  concerning  the  errors  attributable  to  each 
design  decision  can  be  found  in  Appendix  E. 

4 . 2  Overview  of  the  features  of  TOPPS 

TOPPS  is  a  small  expression -oriented  language  designed 
for  teaching  students  the  concepts  of  asynchronous  processes 
and  of  interprocess  communication.  Processes  are  virtual 
processors  executing  procedures  asynchronously  and 
concurrently.  Interprocess  communication  is  accomplished 
through  the  use  of  the  primitives  request  and  release,  which 
operate  on  items  with  a  special  data  type:  resources,  a 
combination  of  semaphores  and  queued  values. 

In  addition,  TOPPS  provides  users  with  operators  for 
assignment  and  for  numerical  and  logical  computations. 
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Character  string  manipulations  are  performed  with  built-in 
functions.  Expression  evaluation  proceeds  from  right  to 
left  with  equal  precedence  among  the  operators. 

Control  constructs  permit  sequencing,  selection,  and 
repetition.  Sequencing  is  indicated  by  separating 
consecutive  executable  expressions  with  a  semicolon.  The 
semicolon  is  used  aS  a  separator;  no  semicolon  may  be  placed 
after  the  final  expression  in  a  compound  expression.  In 
declarations,  on  the  other  hand,  the  semicolon  is  used  as  a 
terminator.  Selection  is  performed  with  an  if  expression, 
which  requires  an  else  clause;  and  repetition  is  performed 
with  a  repeat  expression. 

Compound  expressions  in  procedure  bodies  and  selection 
expressions  must  be  bracketed  using  either  parentheses  or 
begin-end .  However,  if  declarations  are  to  precede  the 
compound  expression,  begin-end  must  be  used. 


Standard 

ALGOL  60 

scope 

rules  are  used 

for  variables. 

Thus,  these 

variables 

are 

not  visible 

outside 

the 

environment 

in  which 

they 

are  declared. 

However, 

the 

variables  may  be  referenced  or  their  names  redeclared  (i.e., 
a  new  variable  with  the  same  name  created)  in  interior 
scopes. 

Sample  TOPPS  programs  appear  in  Appendix  B. 
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4 . 3  The  characteristic  errors  of  ^me  features  of  TOPPS 
4.3.1  Expr^sion  evaluat ion 

The  design  of  the  rule  for  expression  evaluation  in 
TOPPS  was  guided  by  uniformity  and  by  the  treatment  of  the 
assignment  symbol  as  an  "ordinary”  operator.  Expressions 
are  evaluated  from  right  to  left  with  equal  precedence  among 
the  operators  as  in  APL. 

Errors  may  result  from  misconceptions  about  either  the 
direction  in  which  expression  evaluation  proceeds  or  the 
order  in  which  the  operators  are  applied.  If  a  programmer 
were  to  forget  the  direction  in  which  expressions  are 
evaluated^  he  would  not  be  able  to  determine  whether  the 
expression  5-4t3  yields  the  value  -2  (as  it  does  in  TOPPS) , 
or  the  value  4  (as  it  does  in  TOPPSII) .  The  problems  that 
arise  from  the  equal  operator  precedence  of  TOPPS  are 
illustrated  in  the  expression  I+3  <  J*2  in  which  I  and  J  are 
both  integer- valued  variables.  This  expression  does  not 
yield  the  value  true  or  false  depending  on  the  values  of  I 
and  J.  Instead,  it  yields  either  I  or  I  +  1.  If  J  is  less 
than  or  equal  to  one,  the  sub-expression  3  <  (J*2)  yields 
the  value  false  (numeric  zero  in  TOPPS) ,  which  is  then  added 
to  the  value  of  I .  If  J  is  greater  than  one,  the  value  true 
(numeric  one)  is  added  to  the  value  of  I. 

Another  source  of  errors  is  the  parentheses  used  to 
alter  the  strict  lef t-to-right  order  of  evaluation.  A 
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single  parenthesis  may  be  mispunched,  an  extra  parenthesis 
added,  or  a  necessary  parenthesis  omitted  from  an 
expression.  Additionally,  an  expression  may  contain  a 
balanced  number  of  parentheses,  but  one  or  more  of  the 
parentheses  may  be  in  the  wrong  position.  This  error  is 
most  likely  to  occur  in  the  formation  of  a  complex  logical 
condition . 

4.3.2  Assignment 

The  appearance  of  the  assignment  symbol,  :=,  in  an 
expression  in  TOPPS  causes  the  value  of  the  expression  on 
the  right  to  be  stored  in  the  location  specified  on  the 
left,  destroying  the  old  value.  The  operator  produces  a 
value  (the  value  of  its  right  operand)  and  may  be  used 
several  times  in  an  expression.  In  the  following  example,  I 
is  assigned  the  value  3  and  J  is  assigned  the  value  5. 

J  :=  (I  :=  3)  +2 

TOPPS  uses  the  symbol  ;=  for  assignment  and  the  symbol  = 
fcr  comparison  for  equality.  Undoubtedly,  the  major  cause 
of  errors  involving  the  assignment  symbol  is  confusion 
between  the  symbols  :=  and  =.  Many  of  the  subjects  had 
previous  experience  with  languages  thar  use  the  symbol  =  to 
denote  assignment  (e.g.,  FOBTPAN) .  One  might  expect  many 
programmers  to  write  the  expression  I  =  J,  expecting  it  to 
assign  the  value  of  J  to  the  location  specified  by  I  and 
yield  the  value  of  J.  Instead,  I  and  J  are  compared  for 
equality  and  either  the  value  true  or  false  is  returned. 
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A  less  common  but  equally  severe  error  occurs  when  =  is 
replaced  by  ; =  in  a  relation.  In  the  following  example,  the 
body  of  the  repeat  expression  is  not  executed  four  times. 


Instead, 

an 

infinite  loop 

results 

because 

each 

time 

the 

expression 

that  governs 

termination  is 

evaluated , 

I  is 

assigned 

the 

value  four. 

which  is 

always 

false 

in 

TOPPS 

(only  the  rightmost  bit  of  a  value  is  tested) . 

I  :=  0; 
repeat 

I  :=  I  +  1 ; 


until  I  : =  4 ; 


The  ability  to  use  the  assignment  symbol  several  times 
in  an  expression  leads  to  two  other  types  of  errors: 
misunderstanding  about  the  method  of  expression  evaluation 
and  surprising  side  effects.  Errors  of  the  first  type  have 
already  been  discussed  in  Section  4.3.1.  An  error  resulting 
from  an  unexpected  side  effect  is  illustrated  in  the 
following  example,  in  which  the  value  of  the  subscript  of 
the  array  A  is  altered  before  the  value  of  the  element  of 
the  array  is  altered. 

A(I)  :=  I  :=  I  +  1; 


4.3.3  Boolean  operators 


In  TOPPS,  the  operators  8,  1  ,  and  -•  have  two  functions: 
as  connectors  of  logical  relations,  and  as  masking  operators 
to  pack  multiple  values  into  single  words.  Often  these  two 


68 


roles  are  confused.  This  results  in  misunderstanding  the 
value  of  expressions  such  as  the  one  below,  which  is  true 
only  when  the  value  of  I  is  6  (i.e.,  the  "logical  or”  of  the 
two  binary  numbers  2  and  4)  . 

if  I  =  2  I  4  then  . . . 

Errors  may  also  result  from  substituting  the  symbol  \ 
for  the  symbol  &,  and  vice  versa.  Finally,  since  &  and  j 
are  operators,  errors  in  expression  evaluation  similar  to 
those  discussed  in  Section  4.3.1  may  occur  in  expressions 
containing  these  operators. 

4.3.4  Sequencing 

In  TOPPS,  sequencing  of  executable  expressions  is 
indicated  by  separating  consecutive  expressions  with  a 
semicolon.  The  semicolon  causes  the  value  of  the  previous 
expression  to  be  discarded.  All  constructs  in  TOPPS  must 
yield  a  value.  The  value  of  the  re  peat  construct  is  the 
final  value  of  the  expression  executed  in  the  body  of  the 
construct.  The  value  of  the  if  construct  is  the  value  of 
the  expression  executed  in  whichever  of  the  then  and  else 
clauses  is  selected  for  execution.  The  value  of  a 
parenthesized  compound  expression  is  the  value  of  its  last 

sub-expression.  All  of  these  constructs  allow  compound 

/ 

expressions  to  be  embedded  within  them.  In  order  that  such 
constructs  yield  a  value,  no  semicolon  may  follow  the  final 
expression  in  a  compound  expression.  In  the  following 
example.  Index  will  be  assigned  the  position  of  the  element 
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of  Reserved_word  that  matches  Str.  If  Str  does  not  match 
any  of  the  values  in  Reserved_word,  the  value  of  Index  will 
be  zero. 

J  :=  I  :  =  0; 

Index  :=  (repeat 

if  Eeserved_word  (I)  =  Str 
then  J  : =  I 
else  0 ; 

I  ;=”l  +  1 

uni^l  (I  >  #  of_reserved_words)  |  J  -»=  0; 

J)  “ 

Errors  are  caused  by  mispunched,  omitted,  and  extra 
semicolons.  In  particular,  the  addition  of  a  semicolon  to 
the  final  expression  of  a  compound  expression  is  a  frequent 
error. 

In  contrast  to  its  role  as  the  separator  of  expressions, 
the  semicolon  is  used  as  a  terminator  for  declarations.  One 
might  expect  the  users  of  TOPPS  to  make  errors  in  the 
position  of  semicolons  in  declarations  due  to  the 
differences  between  these  uses  of  the  semicolon. 

4.3.5  Selection 

TOPPS  contains  only  one  form  of  the  if  construct:  this 
form  requires  both  then  and  else  clauses.  There  are  many 

t 

errors  attributable  to  this  construct,  only  a  few  of  which 
are  discussed  in  this  section.  A  more  complete  list  of 
errors  can  be  found  in  Appendix  E. 

The  most  common  errors  result  from  the  if  expression  of 
TOPPS  returning  a  value.  Because  a  value  must  be  returned 
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regardless  of  the  value  of  the  condition  in  the  if  clause, 
the  if  expression  must  be  two-tailed  (i.e.,  the  else  clause 
must  always  be  specified)  and  neither  of  the  then  and  e Ise 
clauses  may  be  empty.  In  addition,  the  values  returned  by 
the  then  and  and  else  clauses  should  have  the  same  type. 

Other  errors  are  caused  because  of  the  nesting  structure 
of  the  construct.  The  selection  condition  in  an  outer  if 
expression  might  be  inadvertently  satisfied  or  confusion  may 
exist  about  where  control  proceeds  after  the  execution  of  an 
inner  clause.  Another  error  is  caused  by  a  lack  of  positive 


checking  [Bandell 

1971  ]. 

In 

programming 

with  nested 

if 

statements,  a 

programmer 

tends  to 

specify  all 

the 

interesting  cases 

except 

one. 

The  final  case  is  handled 

by 

the  final  else  clause,  rather  than  by  the  then  clause  of 
another  if  statement.  This  style  of  programming  often  leads 
to  execution  of  the  code  for  the  final  else  clause  when  an 
unexpected  error  occurs.  These  errors  led  to  the  inclusion 
of  a  case  statement  in  TOPPSII.  A  discussion  of  this 
feature  can  be  found  in  Section  4.4.5. 

4.3.6  Bepetition 

In  TOPPS,  repeated  execution  of  a  compound  construct  is 
performed  using  a  repeat  construct.  The  compound  construct 
is  executed  until  the  expression  following  the  reserved  word 
until  becomes  true,  and  is  always  executed  at  least  once. 


71 


As  was  the  case  with  selection  statements,  many  errors 
are  attributable  to  this  construct,  only  a  few  of  which  are 
discussed  in  this  section.  A  more  complete  list  of  errors 
can  be  found  in  Appendix  E. 

Requiring  re  peat  expressions  to  return  values  leads  to 
two  types  of  errors:  failing  to  include  at  least  one 

expression  in  the  body  of  the  construct  and  suffixing  the 
final  expression  with  a  semicolon. 

One  of  the  major  sources  of  errors  in  repeat  expressions 
is  the  control  variable.  Unlike  the  iterative  statement 
shown  below,  the  syntax  of  the  repeat  expression  neither 
requires  the  specification  of  an  initial  value  for  the 
control  variable  nor  provides  for  the  modification  of  the 
variable  during  execution  of  the  construct. 

do  I  =  1  to  1 0  by  1 : 

Another  error  results  from  the  nesting  of  repeat  constructs 
using  the  same  control  variable.  In  the  following  example, 
the  outer  loop  would  be  executed  once  if  N  =  M  1,  and 
unboundedly  otherwise. 

I  :=  0; 

repeat 

9 

9 

T  •  =  0  • 

repeat 

9 

9 

I  :=  I  +  1 

until  I  =  M; 

I  :=  I  +  1 

until  I  =  N 
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Incorrect  specification  of  the  termination  condition  in 
the  until  clause  takes  many  forms.  One  such  error  results 
from  the  incorrect  specification  of  the  bounds  of  an  array 
in  a  repeat  construct  meant  to  be  executed  cnce  for  each 
member  of  an  array.  This  duplication  of  information  invites 
errors. 

The  problems  caused  by  the  errors  involving  the  control 
variable  and  the  specification  of  termination  conditions  led 
to  the  inclusion  cf  the  for  statement  for  arrays  in  TOPPSII. 
A  discussion  of  this  feature  can  be  found  in  Section  4.4.6. 

4.3.7  Statement  brackets 

In  TOPPS,  compound  expressions  in  procedure  bodies  or 
the  then  or  else  clauses  of  if  expressions  must  be  delimited 
either  by  parentheses  or  the  reserved  words  begin  and  end. 
The  reserved  word  until  brackets  compound  expressions  in  the 
repeat  expression.  However,  if  declarations  are  to  precede 
an  expression  in  these  constructs,  begin-end  bracketing  must 
be  employed. 

Ondoubtedly,  most  of  the  errors  concerning  statement 
brackets  are  a  result  of  mispunched,  omitted,  or  extra 
delimiters.  However,  other  errors  may  also  occur,  such  as 
statement  brackets  being  incorrectly  nested.  In  the 
following  example,  the  delimiter  end  should  follow  the 
until . 
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begin 

repeat 

end 

« 

until  . . . 

This  example  includes  another  example  of  useful  redundancy 
(different  reserved  words  being  used  as  closing  delimiters: 
end  and  until)  that  allows  this  error  to  be  detected.  A 
language  such  as  FL/I,  which  closes  all  of  its  compound 
sratements  with  end,  only  determines  that  an  equal  number  of 
opening  and  closing  delimiters  is  present. 

Another  error  occurs  when  declaration  statements  appear 
in  a  compound  expression  bracketed  by  parentheses. 

4.3,8  Inheritance  of  environment 

Standard  ALGOL  60  scope  rules  are  used  for  variables  and 
arrays  in  TCPPS.  Wulf  and  Shaw  have  enumerated  some 
problems  with  the  scope  rules  of  ALGOL  60  as  well  as  several 
attributes  that  a  desirable  alternative  should  possess.  One 
of  these  attributes  is  that  the  current  environment  nor  be 
automatically  extended  to  inner  environments.  The  automatic 
inheritance  of  environment  leads  to  the  two  types  of  errors 
discussed  in  Section  2. 3. 2.1. 
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4.3.9  Constants 

All  constants  in  TOPPS  are  literal  constants  (e.g. , 
and  *abc*)«  same  literal  constant  often  appears  several 

places  in  a  program  (e.g.,  an  array  bound  in  the  relation 
that  controls  the  execution  of  an  iteration  construct) .  If 
all  the  occurrences  of  a  literal  constant  are  to  be  changed, 
each  of  them  must  be  altered  or  an  error  will  occur. 

A  number  of  the  features  of  TOPPS  were  redesigned  and 
included  in  a  new  statement-oriented  dialect  of  the 
language,  TOPPSII.  No  significant  changes  were  made  to  the 
features  that  provide  for  asynchronous  processes  and 
interprocess  communication.  The  number  of  infix  operators 
in  TOPPS  was  reduced  in  the  new  language,  with  assignment 
becoming  a  statement  and  the  Boolean  operators  being 
replaced  by  relation-connecting  functions.  Expression 
evaluation  was  altered  to  proceed  from  left  to  right  with 
traditional  precedence  among  the  operators  (e.g.# 
multiplications  performed  before  additions) . 

As  in  TOPPS,  sequencing  is  indicated  with  the  semicolon. 
However,  the  semicolon  in  TOPPSII  is  a  terminator  rather 
than  a  separator,  appearing  at  the  end  of  each  statement. 
As  in  TOPPS,  the  TOPPSII  if  statement  requires  an  else 
clause.  The  if  statement  has  been  augmented  by  a  case 
statement  in  the  new  language.  A  for  statement  has  also 
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been  added  to  TOFPSII  to  allow  the  programmer  to  specify  an 
action  that  should  be  repeated  for  each  member  of  an  array. 

Statement  lists  in  procedure  bodies,  selection 
statements,  and  for  statements  in  TOPPSII  are  bracketed  by 
the  appropriate  reserved  word  and  end .  The  inclusion  of 
declarations  before  statement  lists  forces  the  introduction 
of  beqin-end  around  declarations  and  statements  only  in 
selection  and  iteration  statements  (where  declarations  are 
not  normally  expected)  . 

The  scope  rules  of  TOPPSII  are  similar  to,  but  simpler 
than,  those  described  in  Section  2. 3. 2.1.  In  each  nested 
procedure,  a  programmer  must  declare  which  (if  any) 
variables  accessible  in  the  immediately  enclosing  procedure 
are  to  be  accessible  in  the  inner  procedure. 

One  other  addition  was  made  in  TOPPSII.  Besides  the 
literal  constants  available  in  almost  every  language, 
TOPPSII  includes  the  ability  to  declare  named  constants  as 
in  Pascal, 

The  reserved  words  used  to  declare  procedures  in  TOPPS, 
program  and  subprogram ,  were  altered  in  TOPPSII  to  parallel 
procedure,  function,  and  procedure.  However,  unlike  the 
previous  changes,  this  change  was  not  expected  to  affect  the 
number  or  severity  of  the  errors  observed. 

Sample  TOPPSII  programs  appear  in  Appendix  B. 
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4.4.1  Expression  evaluation 


Expression  evaluation  in  TOPPSII,  relying  on  its  users* 
previous  experience  with  mathematics  and  other  programming 
languages  (e.g.,  EORTPAN  and  PL/I),  is  performed  from  left 
to  right  with  the  traditional  operator  precedence.  During 
the  design  of  TOPPSII,  it  was  conjectured  that  programmers 
would  make  both  fewer  and  less  severe  errors  using  the 
traditional  method  of  expression  evaluation.  The  examples 
in  Section  4.3.1  now  have  more  traditional  interpretations. 
The  expression  5-4-«-3  has  the  value  4  and  the  expression 
1  +  3  <  J*2  has  the  value  true  if  (1  +  3)  <  (J*2)  and  false 
otherwise. 


4.4.2  Assignment 

As  in  TOPPS,  the  symbol  := 
and  the  symbol  ='  to 
TOPPSII.  However 
operator  in  TOPPSII 
designator  that  may 


is  used  to  specify  assignment 
comparison  for  equality  in 
comparison  symbol  is  an 
symbol  is  a  statement 
per  statement. 


indicate 
while  the 
the  assignment 
appear  only  once 


Since  TOPPSII  is  a  statement-oriented  language,  the 
appearance  of  an  expression  such  as  I  =  J  where  a  statement 
is  expected  leads  to  the  production  of  a  diagnostic  message. 
Similarly,  the  appearance  of  the  assignment  symbol  in  place 
of  an  operator  in  an  expression  like  the  one  below  also 
produces  a  diagnostic  message, 
until  I  : =  4 ; 
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while  the  number  of  errors  attributable  to  confusion  between 


the  two  symbols  was  not  expected  to  decrease  among  the  users 
of  the  new  language,  restricting  the  assignment  symbol  to 
being  a  statement  designator  was  expected  to  reduce  the 
severity  of  the  errors  by  allowing  their  detection  by  the 
compiler.  Errors  resulting  from  confusion  about  the  method 
of  expression  evaluation  or  from  surprising  side  effects 
rely  on  using  the  assignment  symbol  as  an  operator. 
Restricting  the  assignment  symbol  to  being  a  statement 
designator  should  reduce  both  the  number  and  severity  of 
these  errors. 

4.4.3  Ee la ti on-con nec tors 

While  TOPPSII  has  retained  the  operator,  the  &  and  | 
operators  have  been  replaced  by  the  relation-connecting 
functions  all  and  any  respectively.  These  functions  take 
lists  of  arguments,  which  must  be  logical  relations,  and 
return  a  logical  value.  Multiple  values  may  still  be  packed 
inro  a  single  word  using  arithmetic  operators. 

Several  restrictions  on  the  &  and  f  operators  were 
considered  to  prevent  their  misuse. 

1.  Strict  approach:  Allow  only  a  single  operator  (i.e., 
either  S  or  | )  in  separating  two  relations,  in  any 
logical  expression. 

2.  Intermediate  approach;  Allow  only  a  single  all  or  any 
to  group  an  arbitrary  number  of  relations,  but  forbid 
nesting  of  the  functions. 
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3.  Weak  approach;  Allow  arbitrary  nesting  of  the  all  and 

anx  functions. 

The  weak  approach  was  chosen  because  in  resulted  in  the 
relation-connectors  in  TOPPSII  being  as  powerful  as  those  in 
TOPPS .  Either  of  the  other  choices  would  have  forced  the 
users  of  TOPPSII  to  resort  to  nested  if  statements  in 
constructing  complex  conditions. 

Although  more  syntactic  errors  may  occur  because  of  the 
extra  symbols  required  by  the  relaticn-connectors  in 
TOPPSII,  it  was  hoped  that  the  severity  of  the  errors  made 
would  be  reduced.  Separating  the  relation-connecting  and 
bit-packing  roles  of  S  and  j  should  lead  to  fewer 
misunderstandings  about  the  value  of  expressions.  Also,  it 
may  be  easier  to  remember  the  rules  for  evaluating  functions 
(i.e.,  the  innermost  function  first)  than  to  remember  either 
the  rule  for  expression  evaluation  in  TOPPS  or  the  relative 
precedence  of  S  and  f  (if  they  were  to  be  operators  in 
TOPPSII) .  Errors  may  still  result  from  substituting  all  for 
any  and  vice  versa,  but  it  was  felt  that  expressions 
containing  the  relation-connectors  would  be  easier  to  read. 
In  the  following  example,  the  value  true  is  returned  if  the 
already-validated  chess  move  from  Origin  to  Destination  will 
leave  Location  empty.  The  value  false  is  returned 
otherwise.  The  complexity  of  the  expression  is  a  result  of 
provisions  made  fcr  capturing  a  pawn  en  passant .  In  TOPPS 
this  could  be  written  as; 
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return  ((Location  =  Origin) 

I  ((Location  -•=  Destination)  8 
Occupant (Location)  =  Fmpty) 

\  ( (Occupant (Origin)  =  Pawn)  5 

(Occupant (Best inat ion)  =  Empty)  8 
(Origin. File  Destination  .File)  8 

(Location. Fank  =  Origin. Bank)  8 
Location . File  =  Destination. File) ) 

In  TOPPSII^  this  example  could  be  rewritten  as: 

return  (any  of  (Location  =  Origin, 

all  of  (Location  -«=  Destination, 

Occupant (Location)  =  Empty)  , 
all  of  (Occupant  (Origin)  =  Pawn, 

Occupant (Dest ination)  =  Empty, 
Origin. File  -•=  Destination. File, 
Locat ion. Bank  =  Origin. Bank, 

Location . File  =  Destination. File) )) ; 


4. 4. 4  Seguencing 


As  in  TOPPS,  sequencing  in  TOPPSII  is  indicated  by  a 
semicolon.  However,  since  statements  in  TOPPSII  do  not 
return  values,  semicolons  may  appear  after  every  statement, 
rather  than  only  between  expressions.  Although  the  rules  in 
each  language  seem  equally  uniform  and  easy  to  recall,  it 
was  felt  that  fewer  semicolon  errors  would  be  committed  by 
the  users  of  TOPPSII  'because  of  the  similarity  of  the 
language  to  other  statement-oriented  languages  (e.g.,  PL/I). 


Unlike  TOPPS,  the  roles  of  the  semicolon  as  punctuation 
for  executable  statements  and  for  declarations  are  uniform 
in  TOPPSII.  That  is,  the  semicolon  is  a  statement 
terminator  in  both  cases.  One  might  therefore  expect  fewer 
errors  in  the  punctuation  of  declarations  in  TOPPSII  than  in 
TOPPS. 
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4.4,5  Selection 


The  if  statement  of  TOPPSII,  like  that  of  TOPPS, 
requires  both  a  then  and  an  else  clause.  This  statement  is 
subject  to  most  of  the  same  errors  to  which  the  construct  in 
TOPPS  is  vulnerable.  However,  errors  resulting  from  failure 
to  correctly  bracket  statement  lists  in  the  clauses  of  an  if 
statement  should  be  reduced  in  TOPPSIT.  In  TOPPS,  if  an 
expression  list  appears  in  either  clause,  it  must  be 
delimited  by  parentheses.  In  TOPPSII,  the  reserved  word 
else  closes  the  then  clause,  and  the  reserved  word  end 
closes  the  else  clause,  whether  the  clause  is  a  single 
statement  or  a  statement  list. 

TOPPSII  also  contains  a  case  construct  similar  to  that 
of  Pascal  described  in  Section  2.6.1.  This  statement  allows 
the  selection  of  one  of  several  labelled  clauses  for 
execution.  The  clause  selected  is  the  one  whose  constant 
label  has  the  same  value  as  the  selection  expression.  The 
case  statement  has  no  else  clause.  If  none  of  the  clause 
labels  has  the  same  value  as  the  selection  expression,  a 
run-time  diagnostic  is  produced  and  execution  halts. 

The  inclusion  of  a  case  statement  in  TOPPSII  provides  a 
clearer  display  of  source  code  and,  for  this  reason,  should 
reduce  both  the  frequency  and  severity  of  the  errors  caused 
by  nested  if  constructs.  Also,  the  lack  of  an  else  clause 
on  the  case  statement  should  encourage  programmers  to  adopt 
a  positive-checking  style  of  programming. 
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The  case  statement  also  introduces  several  new  types  of 


errors.  In  addition  to  the  expected  syntactic  errors,  the 
value  of  the  selection  expression  may  not  match  any  of  the 
labels.  Other  problems  arise  from  labelling  the  clauses: 
labels  may  be  omitted,  placed  on  the  wrong  clauses, 
duplicated,  not  constants,  or  two  labels  that  are  constants 
may  have  different  types. 

4.4.6  Repetition 

TOPPSII  contains  a  repeat  statement  similar  to  that  of 
TOPPS  and  subject  to  the  same  errors.  TOPPSII  also  has  a 
for  statement,  which  indicates  that  the  enclosed  statements 
should  be  executed  once  for  each  element  of  a  specified 
array.  During  each  repetition,  the  control  variable  is  an 
array  reference  to  a  different  member  of  the  array. 
Operations  on  the  control  variable  are  actually  performed  on 
the  element  of  the  array  referenced  by  the  control  variable. 
Each  for  statement  begins  a  new  unit  of  scope  that 
auromatically  inherits  the  right  to  access  all  data  items 
accessible  in  the  enclosing  environment.  The  control 

variable  is  implicitly  declared  in  this  new  environment  and 
may  not  be  accessed  outside  the  for  statement.  In  the 
following  example,  the  for  statement  is  used  to  initialize 
20  elements  of  the  array  Hand, 
array  Hands  bound  3,4; 

each  Card  of  Hand; 
input  (Card)  ; 
end; 
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The  for  statement  was  included  in  TOPPSII  to  avoid 
errors  resulting  both  from  faulty  specification  of  the 
termination  relation  in  repeat  statements  and  from  the 
nesting  of  repeat  statements  using  the  same  '  control 
variable. 

There  are  seme  disadvantages  to  the  for  statement.  It 
should  result  in  more  syntactic  errors  being  committed,  as 
well  as  errors  from  the  specification  of  data  items  that  are 
not  arrays  as  the  objects  of  repetition.  More  seriously, 
there  is  no  provision  for  terminating  the  iteration  when  a 
condition  is  satisfied,  and  it  is  necessary  to  use  an  extra 
variable  to  obtain  the  index  of  an  array  element  that  meets 
a  particular  condition.  The  search  example  in  Section  4.3.4 
would  be  executed  once  for  each  element  of  Reserved_word, 
even  if  a  match  were  found.  The  variable  Index  would  also 
have  to  be  introduced  to  determine  the  position  of  the 
matching  element. 

array  Reserved_word  bound  #__of_reserved_words ; 

• 

J  :  =  0 ; 

Index  :=  0; 

for  each  Word  of  Reserved_word ; 

if  Word  =  Str  then  Index  :=  J; 

else 

end ; 

J  :=  J  +  1 ; 
end; 
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4.4.7  Statement  brackets 

Parentheses  are  not  used  to  bracket  statements  in 
TOPPSII.  Compound  statements  in  procedure  bodies  are 
delimited  by  the  appropriate  reserved  word  introducing  the 
procedure  and  end,  whether  or  not  the  statements  are 
preceded  by  declarations.  The  reserved  word  else  is  the 
closing  delimiter  of  the  then  clause  and  the  reserved  word 
end  closes  the  else  clause  in  if  statements.  The  appearance 
of  a  label  or  the  reserved  word  end  in  a  case  statement 
closes  the  previous  case  clause.  The  repeat  statement  in 
TOPPSII  is  similar  in  structure  to  that  of  TOPPS.  The  for 
statement  is  closed  by  an  end.  As  in  TOPPS,  when 
declarations  precede  statements  in  selection  or  repetition 
sratements,  begin-end  must  be  used.  The  greater  uniformity 
of  the  bracket  structure  of  TOPPSII  was  expected  to  be  an 
advantage  in  reducing  the  number  of  errors  made. 

4.4.8  Inheritance  of  environment 

Several  alternative  designs  were  considered  for  the 
rules  governing  the  inheritance  of  environment  in  TOPPSII, 
One  alternative,  suggested  by  the  form  of  the  proof  rules 
specified  by  Hoare,  was  to  pass  all  ixems  that  are  global  to 
a  procedure  as  parameters.  This  alternative  was  rejected 
because  it  introduced  the  errors  made  in  dealing  with 
ordered  lists  (i.e.,  the  wrong  number  of  lisx  elemenxs  and 
elements  of  the  list  out  of  order) .  Other  negative  factors 
were  the  storage  cost  of  this  scheme  and  the  restrictions 
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inherent  in  the  parameter  passing  mechanisms  of  the  language 
to  prevent  race  conditions.  (I.e.,  all  parameters  to 
procedures  invoked  concurrently  are  passed  by  value.) 
Another  alternative  considered  was  a  hierarchy  of  data 
similar  to  that  discussed  in  Section  2.3.2. 2.  This  was 
considered  too  radical  a  departure  from  the  rules  of  TOPPS 
to  permit  adequate  assessment  of  the  resulting  errors. 
Finally,  a  hierarchy  of  procedures  similar  to,  but  simpler 
than,  that  described  in  Section  2. 3. 2.1  was  adopted. 

Variables  or  arrays  in  TOPPSII  are  not  automatically 
available  to  interior  procedures.  Only  those  nonlocal  items 
specifically  requested  by  a  programmer  in  a  procedure 
heading  (in  addition  to  data  items  declared  locally)  may  be 
referenced  within  a  procedure.  In  addition,  only  those 
items  accessible  in  the  immediately  enclosing  procedure  may 
be  requested  by  an  inner  procedure.  In  the  following 
example,  the  right  to  access  the  array  Lines  must  be 
requested  in  the  procedure  Reader  so  that  the  procedure 
Get_char  can  also  request  the  right.  Besides  Lines,  the 
inner  procedure  also  must  request  the  right  to  reference 
Line#,  which  is  the  parameter  of  Reader. 
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b eg[in ; 

constant  #_of_lines  ;=  10; 
constant  Line_length  :=  120; 

array  Lines  bound  #_of_lines  -  1^  Line_length  -  1; 

Header  of  Line#  knows  Lines; 

• 

££22§^nre  Get_char  of  Position  knows  Lines, Line#; 
input  (Lines (Line #,  Position)); 
end;  /*  of  procedure  Get_char 
end;  /*  of  procedure  Header  */ 
end;  /*  of  begin  */ 

Failure  to  redeclare  Lines  in  the  nonlocal  variable  list  of 
Get_char  would  result  in  the  ingut  statement  being  tagged 
with  the  following  compile-time  diagnostic. 

ACCESS  DENIED  TO  NONLOCAL  IDENTIFISH;  LINES 
Similarly,  the  failure  to  redeclare  Lines  in  the  nonlocal 
variable  list  of  Header  would  lead  to  the  production  of  the 
following  message  on  the  declaration  of  Get_char  at  compile 
time. 

LINES  NOT  KNOWN  TO  OH  DECLARED  IN  PREVIOUS  PROCEDURE 
Environments  are  automatically  inherited  by  nested  begin 
blocks  as  in  ALGOL  60,  and  constants  need  not  be  redeclared 
to  be  accessed. 

While  the  scope  rules  of  TOPPSII  will  undoubtedly  cause 
more  errors  to  be  committed  than  the  rules  of  TOPPS,  it  was 
hoped  that  the  severity  of  the  two  types  of  errors  discussed 
in  Section  2.3.2. 1  would  be  reduced.  The  scope  rules  of 
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TOPPSII  certainly  do  not  offer  complete  protection.  Errors 
may  result  either  when  an  inner  procedure  both  fails  to 
specify  inheritance  of  a  global  variable  and  reuses  its  name 
in  a  local  declaration,  or  when  an  inner  procedure  specifies 
inheritance  of  a  global  variable  that  it  should  not  be 
allowed  to  access.  However,  these  errors  seem  far  less 
likely  than  the  errors  previously  discussed. 

4.4.9  Constants 

In  addition  to  literal  constants,  TOPPSII  has  a 
statement  that  allows  a  value  to  be  named.  During 
translation,  the  value  is  substituted  for  each  subsequent 
occurrence  of  the  name  in  the  source  code.  Using  named 


constants  instead 

of  literals,  if 

all 

appearances 

of  a 

constant  are  to 

be  changed,  only 

the 

value 

in 

the 

declaration  need 

be  altered. 

Also, 

the 

mnemonic 

significance  of  the 

name  should  lead 

to  programs 

that 

are 

easier  to  read.  It  was  felt  that  named  constants  would 
reduce  both  the  number  and  severity  of  the  errors  occurring 
in  TOPPSII  programs,  despite  the  possibility  of  errors 
occurring  in  the  declaration  for  the  identifier. 

4 . 5  Summary 

The  differences  between  TOPPS  and  TOPPSII  are  summarized 
in  the  following  table.  The  errors  attributable  to  each  of 
these  features  were  the  basis  for  the  nine  hypotheses  of 
this  study.  A  typical  null  hypothesis  would  be  that  the 
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errors  attributabl 
languages  would 
alternative  hypoth 
to  a  feature  would 
programs  than  in 
each  of  the  hypoth 


e  to  the  corresponding  features  in  the  two 
be  equally  frequent  (or  severe) .  The 
esis  would  be  that  the  errors  attributable 
be  less  frequent  (or  severe)  in  TOPPSII 
TOPPS  programs.  The  errors  that  comprise 
eses  can  be  found  in  Appendix  E. 


TOPPS 


TOPPSII 


1.  expression  evaluation 
right  to  left  with 
equal  precedence  among 
operators 

2.  assignment  operator 

3.  logical  operators 
&  and  I 

4.  semicolon  as  separator 

5.  selection  statements: 
if 

6.  repetition  statements: 
repeat 

7.  brackets  used  to  close 
compound  expressions: 
end  and  parentheses 

8.  automatic  inheritance 
of  environment 

9.  constants:  literals 


expression  evaluation 
left  to  right  with 
’’traditional”  operator 
precedence 

assignment  statement 

logical  functions 
all  and  any 

semicolon  as  terminator 

selection  statements: 
if  and  case 

repetition  statements: 
repeat  and  for  each 
(element  of  an  array) 

brackets  used  to  close 
compound  statements: 
end 

inheritance  of  environment 
only  upon  specific  request 

constants:  literals  and 
named  constants 


The 

for  each 


reference  grammar  and  the  grammar  of  the 
of  the  languages  can  be  found  in  Appendix 


CO  mpilers 

F. 
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CHAPTER  FIVE 


RESULTS 


5 . 1  Introduction 

This  chapter  presents  the  results  of  the  study  described 
in  Chapter  Three.  A  summary  of  all  the  errors  observed  and 
the  locations  of  the  best  detection  information  for  these 
errors  is  given  first.  Section  5.3  describes  the  effects  of 
the  skills  and  backgrounds  of  the  subjects  on  the  errors 
observed.  After  this,  the  types  and  frequencies  of  the 
errors  attributable  to  each  altered  language  feature  are 
discussed.  Errors  not  included  in  any  of  the  hypotheses  are 
examined  in  Section  5.5.  Section  5.6  contains  examples  of 
errors  attributable  to  the  backgrounds  of  the  subjects.  The 
impact  of  both  compile-time  and  run-time  diagnostic  aids  is 
described  in  Section  5.7.  Finally,  the  statistically 
significant  results  are  summarized. 

5 . 2  Total  errors 

Ten  of  the  25  subjects  who  started  programming  in  TOPPS 
and  15  of  the  26  who  started  programming  in  TOPPSII  finished 
solutions  to  both  problems.  The  difference  in  the  number  of 
subjects  finishing  both  problems  is  only  mildly 
statistically  significant  (at  the  <20%  level)  using  the  Chi- 
square  test  [Freund  1967],  As  was  explained  in  Chapter 
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Three,  only  the  errors  made  by  those  subjects  who  finished 
both  problems  were  considered. 

No  measure  of  the  programming  ability  of  the  subjects 
who  did  not  finish  both  problems  was  made.  However,  15  of 
the  TOPPS  subjects  and  16  of  the  TOPPSII  subjects  completed 
the  course.  Data  for  three  additional  TOPPS  subjects  and 
one  extra  TOPPSII  subject  was  discarded  because  each  of 
these  subjects  copied  one  of  the  programs.  One  other  TOPPS 
subject  coded  only  one  problem,  and  another  TOPPS  subject 
was  ill  during  the  experiment. 

A  total  of  3937  occurrences  of  1248  errors  were  found 
and  analyzed.  The  averages  (per  programmer  for  the  number 
of  errors,  occurrences,  error  runs,  and  statements  in  both 

f 

final  solutions,  and  per  error  for  persistence)  for  errors 
of  all  types  in  the  two  languages  are  summarized  in  the 
following  table.  Each  average  is  followed  by  its  average 
rank  (e.g.,  when  the  subjects  were  ranked  from  1  to  25 
according  to  the  number  of  errors  they  made,  the  TOPPSII 
group  ranked  an  average  of  1.50  better  than  the  TOPPS 
group) .  Although  the  figures  tend  ro  favor  the  TOPPSII 
subjects,  only  the  number  of  occurrences  is  even  mildly 
statistically  significant.  The  difference  in  the 
persistences  of  errors  were  compared  using  the  large-sample 
comparison  of  means  described  in  Section  3.8.2.  All 
measures  compared  in  this  manner  are  marked  with  an 
asterisk. 
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TOPPS 


TOPPSII 


Level 


Errors 


Occurrences 


Error  runs 


Persistence* 


60.50 (13.90) 
196.00(15.05) 
31.10  (13.65) 
3.24 


42.87  (12.40) 
131  .80  (11,63) 
28.67(12.57) 
3.07 


<20X 


Statements  189.20(12.25)  191.13  (13.50) 

The  variances  of  some  of  the  error  measures  are  shown  in  the 


following  table. 


Errors 


Occurrences 


Error  runs 


TOPPS 


2162 


19321 


276 


TOPPSII 

193 

3434 

151 


Using  the  F«test  [Preund  1967],  the  null  hypothesis  of  equal 
variances  for  the  two  groups  of  subjects  can  be  rejected  in 
favor  of  the  alternate  hypothesis  of  unequal  variances  at 
the  <2%  level  for  two  measures;  errors  and  occurrences. 


The  following  table  summarizes  the  errors  attributable 
to  all  the  language  features  that  were  redesigned.  The 
differences  in  errors  and  occurrences,  which  favor  the 
TOPPSII  group,  are  statistically  significant. 


TOPPS 

TOPPSII 

Level 

Errors 

38.00  (16.65) 

14.  47  (10.  57) 

<5% 

Occurrences 

1  19.00  (16.00) 

51.07  (11.00) 

<5% 

Error  runs 

27.40  (14.45) 

22.  20  (12.03) 

Persists nee* 

3.13 

3.53 
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The  average  number  of  errors  per  program mer  discovered 
at  compile  time  and  at  run  time,  and  the  average  number  of 
undetected  errors  are  presented  below.  The  intention  of  the 
design  of  TOPPSII  was  to  decrease  both  the  number  of  errors 
not  detected  until  run  time  and  the  number  of  undetected 
errors.  Again,  although  these  figures  tend  to  favor 
TOPPSII,  none  are  statistically  significant. 

TOPPS  TOPPSII  Level 

Compile  time  39.30(13.80)  26.  00  (12.47) 

Pun  time  15.60  (14.20)  12.  27  (12.  20) 

Undetected  5.60(14.25)  4.60(12.17) 

Eight  of  the  48  undetected  errors  in  TOPPS  programs  and  16 
of  the  69  undetected  errors  in  TOPPSII  programs  remained  in 
the  final  (submitted)  solutions  to  the  problems.  A  more 
detailed  breakdown  of  these  figures  may  be  found  in 
Appendix  G. 

5 . 3  Effects  of  skill  and  background 

In  Section  3.4,  it  was  suggested  that  systematic 
selection  of  subjects  throughout  the  class  might  lead  to  two 
groups  with  differing  abilities.  In  particular,  it  was  felt 
that  graduate  students  might  have  more  programming 
experience  than  undergraduates,  and  full-time  students  might 
have  more  time  to  devote  to  the  problems  than  part-time 
students.  The  following  figures  indicate  that  the 
undergraduates  were  better  programmers.  The  differences  in 
occurrences  and  error  runs  favor  the  undergraduate  group  and 
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are  mildly  significant.  A  further  breakdown  of  the  data 
showed  that  the  undergraduates  were  more  successful 
programmers  in  each  language.  Similar  results  were  obtained 
for  full-time  students.  The  differences  in  errors  and 
occurrences  favor  the  full-time  group  and  are  mildly 
significant.  However,  within  the  TOPPS  group,  the  part-time 
students  had  fewer  error  runs  than  the  full-time  students. 
Part-time  students  also  used  significantly  fewer  statements 
in  their  final  solutions  to  problems. 


Graduates  and  Undergraduates 
both  languages 


Errors 
Occurrences 
Error  runs 
Persistence* 
Statements 


Grad  (18) 

52.44 (13.64) 
163.89  (13.78) 
31.11  (13.78) 
3.13 

194.28  (13.39) 


Undergrad  (7)  Level 

43.43  (1  1  .36) 
141.00(11.00)  <20% 

25.  86  (11.00)  <20% 

3.25 

180.29(12.00) 


Part-time  and  Full-time  Students 
both  languages 


Errors 
Occurrences 
Error  runs 
Persistence* 
Statements 


Part-time  (13) 
58.23  (14.23) 
184.85  (14.73) 
30.54  (13.58) 
3.17 

170.38  (9.38) 


Full- time  ( 1 2)  Level 
40.92  (11  .67)  <20% 

127.83  (11.  13)  <20% 

28.67  (12.38) 

3.12 

212.00  (16.92)  <5% 
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It  could  not  be  predetermined  whether  a  subject's  industrial 
experience  would  be  a  boon  or  a  bane.  Within  each  language, 
comparing  the  subjects  with  any  industrial  experience  to 
those  with  no  industrial  experience  yielded  significant 
differences  only  in  persistences  and  statements  used  by 
TOPPSII  subjects.  It  is  interesting  to  note  that  the 
measures  for  three  of  the  subgroups  are  similar,  and  that 
the  bulk  of  the  errors  lie  in  the  group  of  eight  TOPPS 
subjects  who  had  industrial  experience. 


Experience  and 

No  experience 

TOPPS 

Experience  (8) 

No  experience  (2) 

Level 

Errors 

68.25  (6.00) 

29.  50  (3.50) 

Occurrences 

213.  88  (5.88) 

124.50(4.00) 

Error  runs 

31.63  (5.62) 

29.  00  (5.00) 

Persiste  nee* 

3.13 

4.22 

<20% 

Statements 

196.38  (5.  86) 

160.50 (4.00) 

TOPPSII 

Experience (9) 

No  experience (6) 

Level 

Errors 

42.22  (7.89) 

43.83(8.  17) 

Occurrences 

138.78  (8. 39) 

121.33  (7.42) 

Error  runs 

29.00  (8.1  1) 

28.  17(7.83) 

Persistence* 

3.29 

2.77 

<10% 

Statements 

165.56  (5.  00) 

229.50 (12.50) 

<5% 

A  profile  of 

the  error  measures  for  each  student 

appears 

Appendix  H. 


94 


5 . 4  Evaluation  of  individual  design  decisions 


Although  an  attempt  was  made  to  redesign  mutually 
independent  language  features,  it  is  difficult  to  discount 
the  possibility  that  errors  attributable  to  one  feature  are 
not  related  to  the  errors  attributable  to  another  feature 
(e.g.,  errors  in  expression  evaluation  and  errors  in 
relation-connectors)  .  Thus,  the  individual  comparisons  of 
the  following  two  sections  may  not  be  as  significant  as 
indicated.  An  experiment  with  several  different  languages 
would  be  required  to  factor  out  dependencies. 


5.4.1  Semicolons 


As  was  expected,  fewer  errors,  occurrences,  and  error 
runs  attributable  to  this  design  decision  appeared  in 
TOPPSII  programs.  These  differences  were  all  highly 
significant.  The  difference  in  persistences  (which  were 
small  in  both  languages)  was  not  significant. 


Errors 
Occurrences 
Error  runs 
Persistence 


TOPPS  TOPPSII  Level 

20.20  (17.95)  2.13(9.70)  <,5% 

28.50  (18.  10)  2.  47(9.60)  <.5% 

8.20  (18.40)  1  .73(9.40)  <,5% 


1.41  (119.  20)  1.16(106.75) 


The  202  errors  occurring  in  TOPPS  programs  were  almost 
evenly  divided  between  omitted  semicolons  (110)  and  extra 
semicolons  on  final  expressions  (91) .  One  other  semicolon 
error,  replacing  a  comma  with  a  semicolon  in  a  parameter 
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list,  was  detected  at  run  time  by  a  special  instruction  that 
determines  if  the  number  of  formal  and  actual  parameters 
correspond. 

process  Player  of  0  producing  Mutex__pile_zero ; 

Mutex_pile_one; 

Because  TOPPS  is  an  expression-oriented  language,  this 
single  construct  was  interpreted  (without  warning)  as  a 
compound  construct,  the  second  part  of  which  was  simply  the 
identifier  Mutex_pile_one . 

Twenty-one  of  the  32  TOPPSII  errors  involved  missing 
semicolons.  Of  the  remaining  11  errors,  10  were  caused  by 
the  addition  of  extra  semicolons  and  the  other  by  a 
keypunching  mistake.  The  extra  semicolons  occurred  in  rwo 
locations  in  if  statements:  between  the  selection 
expression  and  the  reserved  word  then,  and  between  the 
reserved  words  else  and  end  (i.e.,  as  the  null  statement). 
One  of  the  design  decisions  of  TOPPSII  was  to  make  the  null 
statement  empty  rather  than  a  semicolon.  Therefore,  the 
appearance  of  a  single  semicolon  between  the  words  else  and 
end  resulted  in  a  diagnostic  message  from  the  parser.  This 
design  decision  was  responsible  for  9  of  the  10  errors 
involving  extra  semicolons,  and  partially  responsible  for  a 
severe  error  in  the  design  of  the  case  statement,  as  will  be 
seen  in  Section  5.4.7. 2. 

Both  languages  use  the  semicolon  as  a  declaration 
terminator.  It  was  expected  that  fewer  errors  would  be  made 
by  TOPPSII  subjects,  since  the  use  of  the  semicolon  in 
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declarations 

and  executable 

statements  in 

TOPPSII  is 

ident ical . 

This  proved  to 

be  the  case. 

as  only  the 

difference  in 

persistences  was 

not  significant. 

TOPPS 

TOPPSII 

Level 

Errors 

2.60(16.50) 

.  40  (10. 67) 

<5% 

Occurrences 

3.40(16.60) 

. 40 (10. 60) 

<5% 

Error  runs 

1.90  (16.75) 

.  20  (10.50) 

<5% 

Persists  nee 

1.31  (17.08) 

1.00  (14.00) 

Of  the  errors  attributable  to  this  language  feature^  25  of 
the  26  errors  in  TOjPPS  programs  and  all  six  of  the  errors  in 
TOPPSII  programs  were  a  result  of  omitted  semicolons. 

Because  TOPPS  is  expression-oriented  and  TOPPSII  is 
statement-oriented,  the  issue  of  semicolon  placement 
involves  both  syntactic  and  semantic  considerations.  The 
purely  syntactic  factor  could  be  investigated  using  two 
statement-oriented  languages,  one  of  which  uses  the 
semicolon  as  a  separator,  and  the  other  of  which  uses  the 
semicolon  as  a  terminator. 

5.4.2  Expression  evaluation 

Errors  due  to  expression  evaluation  result  from 
misconceptions  about  both  the  direction  in  which  it  proceeds 
and  the  order  in  which  the  operators  are  applied.  As  was 
conjectured,  both  fewer  and  less  severe  errors  were  found  in 
TOPPSII  programs.  Because  no  errors  attributable  to  the 
rule  of  expression  evaluation  occurred  in  TOPPSII  programs, 
the  persistence  of  these  errors  could  not  be  computed. 
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TOPPS 

TOPPSII 

Level 

Errors 

.50(16.75) 

.00  (10.50) 

<5% 

Occurrences 

4.30(16.75) 

.  00  (10.50) 

<5% 

Error  runs 

4.  30(16.75) 

.00  (10.50) 

<5% 

Persists nee 

8.60 

— 

Two  of  the 

five  errors 

located  were  due  to 

the  direction 

of  expression 

evaluation. 

That  is,  the 

expressions 

contained  only  operators  that  would  have  had  the  same 
precedence  if  the  operators  had  any  precedence-  The  first 
of  these  errors  resulted  from  the  rightmost  subtraction 
being  performed  first. 

Available  :=  Max_available_buff ers 

-  Buf f ers_allocated_output 

~  Buffers_allocated_input_s pooler 

After  several  runs  that  produced  increasing  amounts  of 
intermediate  output,  the  subjecr  located  the  error  and 
rewrote  the  expression  in  the  following  manner. 

Available  :=  Hax_available__buf f ers 

-  Buf f ers_allocated_output ; 

Available  :=  Available  -  Buf f ers_allocated_input_£pool€r 
The  second  error  was  similar,  but  involved  both  the 
subtraction  and  addition  operators.  In  the  argument  that 
determines  the  starting  position  of  the  substring  to  be 
selected,  the  addition  was  performed  before  the  subtraction. 

I  :=  1; 

Drac  :=  *  ’ ; 

repeat 

Drac  ;=  cat  (Drac,  substr  (Card.  Nochar-I-*-1 ,  1)) 
until  Nochar  <  I  ;=  I  1 
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This  error  was  detected  by  the  interpreter,  which  complained 
that  the  second  argument  to  the  substring  function  was 
negative. 

The  three  errors  that  were  caused  by  misconceptions 
about  the  precedence  of  operators  each  involved  a  subject's 
expectation  that  certain  operators  had  precedence  over  other 
operators.  These  errors  were  aggravated  by  the  implicit 
conversion  of  logical  values  to  integers  (as  also  occurs  in 
APL)  .  In  the  case  of  the  first  error,  a  subject  expected 
the  arithmetic  operators  to  have  precedence  over  the 
relational  operators. 

Maxoutlines  ;=  4; 

I  :=  1; 

repeat 

release (Okreader) 

until  Bufsize  -  Maxoutlines 

Thus,  he  expected  the  expressions  cn  either  side  of  the 
relational  operator  (i.e. ,  <)  to  be  evaluated  before  the 
relational  operator  was  applied. 

The  second  error  was  also  the  result  of  a  subject's 
expectation  that  arithmetic  operators  had  precedence  over 
relational  operators. 

if  (Number_on  (Queue)  <  Number)  \ 

{  (Proc_id  =  Inspool_id)  S 

Number_on  (Queue)  Number_potentially__available  <  = 
Max_required_by__a_cpu) )  then  . . . 

In  this  example  it  is  obvious  that  the  subject  understood 

the  direction  of  expression  evaluation,  since  he  realized 

that  the  first  relation  needed  to  be  parenthesized  in  order 

that  the  result  of  the  comparison  (rather  than  the  value  of 
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Number)  was  the  left  operand  of  However,  he  failed  to 

parenthesize  the  expression: 

Number_on  (Queue)  +  Nuffiber_potentially__available 
to  make  it  (rather  than  the  value  of 

Number__potentially_available)  the  left  operand  of  <=. 

The  final  error  appeared  because  a  subject  expected  the 
relational  operator  =  to  have  precedence  over  the  Boolean 
operator  &.  The  following  expression  was  to  yield  the  value 
true  if  the  four  elements  of  the  array  Card  contained  the 
same  positive  integer. 

if  (Card(l)  =  Card  (2)  S  Card(1)  =  Card  (3)  S 
Card(1)  =  Card  (4))  then  ... 

Instead,  because  each  of  the  relations  was  not 

parenthesized,  the  expression  yielded  the  value  true  when 
the  value  of  Card(1)  was  0,  the  value  of  Card  (2)  was  an  even 
integer,  and  the  value  of  Card  (4)  was  a  non-zero  integer. 
Under  these  conditions,  the  value  of  Card  (3)  was  irrelevant. 

5.4.3  Relation-connectors 

The  programs  of  each  group  of  subjects  contained  only 
two  errors  attributable  to  this  design  decision.  It  appears 
that  the  greater  number  of  errors  expected  in  TOPPSII 

programs  did  not  materialize.  Furthermore,  as  hypoxhesized, 
the  difference  in  persistences  favors  the  TOPPSII  group,  but 
is  only  mildly  significant  because  of  the  small  number  of 
errors  observed. 
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TOPPS 


TOPPSII 


Level 


Errors 
Occurrences 
Error  runs 
Persistence 


.  20(13.50) 
2.30(13.70) 
2.30(13.70) 
11,50  (3.50) 


.  13(12.67) 

.  13(12.53) 

.  13 (12. 53) 

1  .00(1.50)  <20% 


The  only  errors  observed  in  TOPPS  programs  were  those 
involving  misconceptions  about  the  order  of  expression 
evaluation.  The  two  expressions  containing  Boolean 
operators  in  which  this  problem  arose  have  been  discussed  in 
the  previous  section.  The  TOPPSII  errors  involved  omitting 
the  reserved  word  of  and  adding  an  extra  comma  to  the  list 
of  relations. 

Thirteen  occurrences  of  eight  errors,  resulting  from  an 
error  in  the  implementation  of  relation-connectors  in  the 
TOPPSII  compiler,  were  not  included  in  the  above  figures. 
Expressions  such  as  the  one  shown  below  returned  the  value 
false  in  TOPPSII  when  the  low  order  bit  of  K  was  zero. 

all  of  (I  <  J,  K) 

The  TOPPSII  subjects  were  issued  a  grammar  update  which 
required  <logical  expression>s,  rather  than  <expressions>s 
as  the  arguments  of  relation-connectors.  After  this  change, 
the  parser  emitted  an  error  message  for  expressions  like  the 
one  above. 
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5.4.4  Assignment 


The  TOPPS  subjects  performed  slightly  more  assignments 
than  the  TOPPSII  subjects  did.  However,  the  difference  in 
the  average  of  the  maximum  number  of  assignments  occurring 
during  the  solution  of  a  problem  (37.80  in  TOPPS  and  35.47 
in  TOPPSII)  was  not  significant.  One  explanation  for  the 
greater  number  of  assignments  in  TOPPS  programs  is  that 
TOPPS  subjects  wanted  to  use  short  expressions  to  assure 
themselves  of  the  correctness  of  their  computations. 

All  the  observed  errors  attributable  to  this  design 
decision  were  the  result  of  substituting  the  assignment 
symbol  for  the  equality  symbol  and  vice  versa.  While  the 
numbers  of  errors  of  this  type  were  not  expected  to  decrease 
among  the  users  of  TOPPSII,  restricting  the  assignment 
symbol  to  a  statement  designator  allowed  the  production  of 
diagnostic  messages  that  were  expected  to  reduce  the 
persistence  of  errors.  The  differences  in  errors  and 
occurrences,  while  large,  are  not  statistically  significant 
because  they  represent  the  contribution  of  a  small  number  of 
subjects  who  made  a  large  number  of  mistakes.  The  measures 
of  severity  favor  the  TOPPSII  group  and  bear  out  the 
hypothesis.  The  difference  in  error  runs  is  mildly 
significant,  and  the  difference  in  persistences  is  highly 
significant. 
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TOPPS 


TOPPSII 


Level 


Errors 


2.  30(14.35) 


80  (12. 10) 


Occurrences 


16.40  (14.  95)  1.13(11.70) 


Error  runs 


7.00(15.15) 


80  (11.57) 


<20% 


Persistence  7.13(23.17) 


1  .42(8.08) 


<.1% 


In  the  following  attempt  to  convert  a  character  to  an 
integer  in  TOPPS^  the  equality  operator  incorrectly  replaced 
the  assignment  operator  three  times^  causing  errors  that 
manifested  themselves  only  in  the  output  of  the  program, 
if  First  char  =  *  1 • 


then  Quantity  ;=  1 
else  if  First_char  =  *2* 

then  Quantity  :=  *2* 
else  if  First__char  =  *3* 
then  Quantity  =  3 
else  if  First^char  =  *4* 
then  Quantity  =  4 
®ise  Quantity  =  0 


The  assignment  of  ’2*  to  Quantity  was  also  an  error,  but  it 
is  not  relevant  to  this  discussion.  In  TOPPSII,  each  of  the 
three  source  lines  in  error  would  be  followed  by  a  message 
from  the  parser  indicating  that  a  <storage  reference>  may 
not  be  followed  by  the  equality  symbol.  Eighteen  of  the  23 
errors  in  TOPPS  and  three  of  the  12  errors  in  TOPPSII  were 
the  result  of  replacing  an  assignment  symbol  with  an 
equality  symbol. 

The  complementary  error,  replacing  an  equality  symbol 
with  an  assignment  symbol,  caused  errors  that  manifested 
themselves  in  several  manners.  In  the  following  example, 
this  error  caused  two  processes  (Inspooler  and 
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Output__spooler)  to  terminate  prematurely.  In  Inspooler,  the 
value  of  True  (a  global  variable  containing  the  integer 
value  one)  was  assigned  to  Flag  and  returned  as  the  value  of 
the  expression  controlling  execution  of  the  repeat 
expression. 

program  Inspooler  . . . 
repeat 

until  Flag  :=  True 

program  Cpu  ... 
repeat 

• 

until  Cpu_activity  ;=  False 

end; 

prog  ram  Output_spooler  ... 
repeat 

until  Output_activity  :=  True 

end; 

Eventually,  the  processes  deadlocked  because  the  Inspooler 
and  Output_spooler  finished  execution,  and  the  Cpu  waited 
for  a  resource  produced  by  the  Inspooler.  This  condition 
was  detected  by  the  interpreter  and  the  program  was 
terminated.  In  TOPPSII,  each  of  the  three  source  lines  in 
error  would  be  followed  by  a  parser  message  indicating  that 
the  assignment  symbol  may  not  appear  in  an  expression.  Five 
of  the  23  errors  in  TOPPS  programs  and  nine  of  the  12  errors 
in  TOPPSII  programs  were  the  result  of  replacing  an  equality 
symbol  with  an  assignment  symbol. 
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5.4.5  Inheritance  of  environment 


While  it  was  expected  that  the  scope  rules  of  TOPPSII 
would  cause  more  clerical  and  syntactic  errors  to  be 
committed  than  the  rules  of  TOPPS,  it  was  hoped  that  the 
severity  of  errors  that  could  not  be  detected  syntactically 
would  be  reduced.  As  expected,  the  number  of  errors  favored 
the  TOPPS  group  and  was  significant,  and  the  persistence  of 
errors  favored  the  TOPPSII  group  and  was  highly  significant. 
As  it  could  not  be  determined  before  the  experiment  whether 
the  number  or  the  severity  of  the  errors  would  dominate  the 
differences  in  occurrences  and  runs,  a  two-tailed  test  was 
used  to  compare  these  differences.  The  TOPPSII  group  had 


fewer  average 

occurrences. 

but  they  involved 

more 

programmers. 

so  the  average 

rank  of  the  TOPPSII  group 

was 

significantly 

greater  than  that 

of  the  TOPPS 

group . 

The 

difference  in 

runs  also  favored  the  TOPPS 

group  and 

was 

significant. 

TOPPS 

TOPPSII 

Level 

Errors 

.  40(8.90) 

1  .93(15.73) 

<5% 

Occurrences 

3.90 (9.90) 

3.27(15.07) 

<10% 

Error  runs 

1.90(9.90) 

2.27  (15.07) 

<10X 

Persistence 

9.75(30.75) 

1  .69(15.10) 

<.5X 

The  design  decision  is  more  easily  tested  by  comparing 
only  the  errors  that  were  not  detected  syntactically.  In 
this  case  all  the  figures  favor  the  TOPPSII  group,  but  only 
the  difference  in  persistences  is  mildly  significant. 
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TOPPS 


TOPPSII 


Level 


Errors 
Occurrences 
Error  runs 
Persistence 


.40  (13,55) 
3.90(13.60) 
1.90(13.60) 
9.75(4.75) 


.  20  (12.63) 
1.33(12.60) 
.  80  (12.60) 

6  .67(3.00) 


<20% 


One  of  the  errors  in  the  TOPPS  programs  was  the  result 
of  a  misspelled  formal  parameter  (Dard4)  causing  a  reference 
to  it  (Card4)  to  be  bound  to  a  global  data  item, 
variable  Cardi,  ...f  Card 4; 


program  Player  of  Playerid,  Cardi,  ...,  Dard4  ...  is 
begin 

9 

Card (4)  :=  Card4; 


end; 

The  other  three  errors  resulted  from 
declarations,  and  having  references  bound 
items.  This  problem  is  especially  acute 
features  that  support  concurrency. 


omitting  local 
to  global  data 
in  languages  with 
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As  is  the  case  with  languages  that  do  not  support 

concurrency,  two  separate  procedures  may  alter  the  same 
global  data  item  (e.g«,  I  in  the  program  shown  below).  In 
addition,  languages  supporting  concurrency  allow  a  procedure 
to  be  invoked  more  than  once  as  a  process.  Therefore,  even 
a  single  procedure  altering  a  global  data  item  (e.g.. 
Buff er_pointer  in  the  program  shown  below)  may  lead  to  a 
hard- to-detect  error  when  the  procedure  is  invoked  twice, 
variable  I ; 

variable  Buff er_pointer ; 
program  Input_spooler  . . . 

• 

« 

T  1  . 

-1.  .  If 

repeat 


until 


2  <  I 


1+1; 


end; 

program  Cpu  ... 

a 

repeat 

Buffer_pointer  :=  request (Input  buffer)  ; 


until  . . . ; 


repeat 

release  (Output  operation)  ; 
iatil  K  < '  I  :  =”'l  +  1 ; 


end; 

/*  mainline  '^ / 

I  :=  1; 
repeat 

release  (Avail,  I) ; 
until  Bufsize  <  I  :=  I  +  1; 
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As  expected,  most  of  the  errors  attributable  ro  the 
inheritance  of  environment  in  TOPPSII  programs  were  of  the 
clerical  or  syntactic  variety.  Six  errors  involved  the 
reserved  word  knows  or  the  commas  that  punctuate  the 
nonlocal  variable  list.  Sixteen  errors  involved  failures  to 
include  subsequently  referenced  identifiers  in  nonlocal 
variable  lists.  One  of  these  errors  prompted  a  subject  to 
correctly  move  the  declaration  of  a  data  item  inside  the 
procedure  in  which  it  was  referenced.  The  other  15  errors 
resulted  in  additions  of  identifiers  to  nonlocal  variable 
lists.  Two  other  errors  were  caused  by  the  inadvertent 
redeclaration  of  names  already  declared  in  the  same  scope. 

The  decision  to  provide  for  inheritance  of  only  one 
level  led  to  the  detection  of  one  potentially  severe  error. 
In  the  procedure  Strategy,  a  subject  meant  to  acquire  the 
right  to  access  the  array  Hnd  declared  in  Player,  but 
instead  acquired  the  right  to  access  the  array  Hand  declared 
globally,  but  accessible  to  Player.  An  error  message  was 
produced  when  the  innermost  procedure,  Find_pair,  attempted 
to  acquire  access  rights  to  Hnd,  which  was  not  accessible  in 
the  immediately  enclosing  procedure.  Strategy. 
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•  •  • 


knows  Hand 


array  Hand  bound  4,4; 
parallel  procedure  Player  of 
array  Hnd  bound  5 ; 


parallel  procedure  Strategy  of  ...  knows  Hand,  ...; 


parallel  procedure  ?ind_pair  of  . . . 
knows  Hnd,  Pair; 


The  scope  rules  of  TOPPSII  do  not  provide  complete 
protection.  After  being  denied  the  right  to  access  a  global 
data  item  (End__sw)  ,  a  subject  chose  to  redeclare  the  item 
within  the  procedure  in  which  it  was  referenced,  rather  than 
acquiring  the  right  to  access  the  global  item.  Two 
processes  were  to  execute  the  procedure  User  until  one  of 
the  processes  set  the  global  End_sw  to  one.  Instead,  one 
process  terminated,  and  the  other  process  continued  to  wait 
for  an  unavailable  resource. 

variable  End^sw; 

parallel  procedure  User  of  Cpu  producing  Lines 
knows  Buffer_size; 
variable  End_sw ; 

repeat; 

until  End_sw  =  1 ; 

end ; 

End_sw  ;=  0; 

Eventually,  the  subject  moved  the  identifier  End__sw  into  the 
nonlocal  variable  list  and  the  resulting  error  message  led 
him  to  correct  his  program. 
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Another  subject  chose  to  acquire  access  rights  to  two 
global  data  items  (Indx1  and  Indx2)  that  were  used  as  loop 
control  variables  in  a  procedure  that  was  invoked  five  times 
as  a  process. 

variable  Indx1,  Indx2; 

parallel  procedure  Player  of  ...  knows  . . . ,Indx1 , Indx2; 

• 

Indx1  :=  0; 

repeat; 

output  ( ‘card  * ,Indx1 , *  is • , Hands (Number , Indxl ) )  ; 

Indxl  :=  Indxl  +  1; 

until  Indxl  >  3; 

« 

Indx2  ;=  0; 

^£eat ; 

• 

• 

Indx2  ;=  Indx2  +  1; 

until  Indx2  >  3; 

end; 

These  errors  were  eventually  detected  by  the  interpreter 
because  of  a  subscript  exceeding  the  declared  bounds  of  an 
array  in  an  output  statement. 

The  diagnostics  associated  with  the  scope  rules  in 
TOPPSII  also  served  as  the  manifestations  of  other  errors. 
For  example^  a  subject  was  denied  access  tc  a  global  data 
irem  in  an  expression  and  realized  that  he  was  using  the 
wrong  identifier  in  rhe  expression. 
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5.4.6  Statement  brackets 


The  greater  uniformity  of  the  bracket  structure  of 


TOPPSII  was 

expected  to  be  an 

advantage 

in  reducing 

the 

number  of 

errors  made  by 

the  subjects 

working  in 

this 

language . 

The  differences 

in  all  the 

measures 

were 

significant. 

TOPPS 

TOPPSII 

Level 

Errors 

1 .40(16.00) 

• 

• 

o 

o 

<5% 

Occurrences 

2.  30  (16.  90) 

,53  (10.40) 

<5% 

Error  runs 

1  .70(16.75) 

.  53  (10.50) 

<5% 

Persistence 

1 .64  (12.54) 

1.14(7.93) 

<10% 

Of  the  14  errors  in  TOPPS  programs,  half  were  due  to  the 
reserved  word  end  being  incorrectly  added  or  omitted.  Three 
other  errors  involved  the  use  of  parentheses  to  enclose 
statement  lists  containing  declarations.  More  subtle  errors 
also  occurred.  One  example  of  such  an  error  was  caused  by 
allowing  the  else  clause  of  an  if  construct  to  contain 
either  a  single  expression  or  a  delimited  compound 
expression,  rather  than  simply  requiring  a  bracket  in  both 
cases.  In  the  following  example,  this  design  decision 


allowed  missing  parentheses  to  go  undetected 


if  . . .  then  ( 

request (Spoon)  ; 
if  No_of_spoons  <  1  then  ( 

Spoon_received (Gambler_id)  ;=  Nil; 
output  (’player  Gamble r_id, 

*  has  lost  the  game*)) 
else  No_of_spoons  :=  No_of__spoons  -  1; 

Spoon_received  (Gambler_id)  ;=  Yes; 
output (* player  * , Gambler_id, 

»  managed  to  get  1  spoon*); 
release (Spoon) ) 
else  0 

Parentheses  should  have  been  placed  before  the  construct 
that  decremented  No_of __spcons  and  after  the  following  out  put 
construct. 


Six  of  the  seven  errors  in  TOPPSII  programs  were  due  to 
the  reserved  word  end  being  incorrectly  added  or  omitted. 


5.4.7  Additional  statements 


The  results  of  providing  constant  identifiers  and  case 
and  for  statements  were  inconclusive  for  two  reasons. 
First,  the  hypotheses  concerning  the  relevant  errors  were 
too  broad  (e.g.,  any  error  in  selection).  Also,  substitutes 
for  these  constructs  were  available  in  both  languages  and 
were  used  heavily,  while  the  new  constructs  were  not 
frequently  used  in  TOPPSII.  One  possible  explanation  for 
the  failure  to  use  the  for  statement  was  the  lack  of  an  exit 
statement  in  TOPPSII.  Subjects  seemed  reluctant  to  perform 
the  remainder  of  an  iteration  after  a  condition  was 
satisfied.  Below  is  a  table  summarizing  the  average  number 
of  the  different  types  of  statements  used  on  each  run. 
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TOPPS 


TOPPSII 


t  an  t  1.79 

if  8.36  7.76 

case  0.50 

repeat  7.76  6.15 

for  0.40 


5. 4. 7.1  Constants 


It  was  expected  that  the  addition  of  constant 
declarations  would  reduce  both  the  number  and  severity  of 
errors  occurring  in  TOPPSII  programs.  As  the  figures  below 
indicate,  there  were  fewer  errors  and  occurrences  in  TOPPS 
programs,  but  the  average  rank  of  the  occurrences  in  TOPPSII 
programs  is  lower.  Fewer  runs  were  ruined,  and  errors  had 
lower  persistences  in  TOPPSII  programs. 


Errors 
Occurrences 
Error  runs 
Persistence 


TOPPS 

2.30  (12.  35) 
12.80  (13. 50) 
9.50(13.50) 
5.57(42.59) 


TOPPSII 
3.20(13.43) 
13.  33  (12.67) 
8.40  (12.67) 
4.17(32.84) 


Level 


<5% 


Twenty  of  the  48  errors  in  TOPPSII  programs  were 
directly  attributable  to  the  declaration  of  constants.  Four 
of  these  errors,  each  with  a  persistence  of  one  run, 
occurred  when  a  subject  failed  to  terminate  a  constant 
declaration  with  a  semicolon.  The  remaining  16  errors  were 
caused  by  the  scope  rules  of  TOPPSII  which  consider 
constants  to  be  global  data  items.  That  is,  they  need 
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neither  be  redeclared  nor  included  in  nonlocal  variable 
lists  to  be  accessible  to  inner  procedures.  Half  of  these 
errors,  each  with  a  persistence  of  one  run,  resulted  from 
redeclaring  constants  in  inner  procedures.  The  other  half 
of  the  errors,  with  an  average  persistence  of  1.63  runs, 
occurred  when  a  subject  included  a  constant  identifier  in  a 
nonlocal  variable  list.  Removing  these  errors  causes  the 
number  of  errors  and  occurrences  in  TOPPSII  programs  to  be 
slightly  less  than  those  in  TOPPS  programs.  Also,  the 
significance  of  the  difference  in  persistences  disappears. 


TOPPS  TOPPSII  Level 

Errors 

2.30  (13.95)  1.87(12.37) 

Occurrences 

12.80  (13.90)  1  1.67  (12.40) 

Error  runs 

9.50(13.95)  7.73(12.37) 

Persistence 

5.57  (24.98)  6,25(26.84) 

5. 4. 7. 2  Selection  statements 

The  case  statement  was  added  to  TOPPSII  to  reduce  the 
number  and  severity  of  errors  attributable  to  selection 


statements. 

The  differences  in  errors  and  occurrences 

favored  the 

TOPPSII  group  and  were  significant.  The 

differences 

in  error  runs  and  persistences  both  favored  the 

TOPPS  group,  but  were  not  significant. 
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TOPPS 

TOPPSII 

Level 

Errors 

9.60  (15.70) 

5.40(1  1.20) 

<10% 

Occurrences 

35.00(14.55) 

23.80  (11.96) 

<20% 

Error  runs 

13.60 (12.70) 

15. 33 (13. 20) 

Persistence 

3.65(87.72) 

4.41  (90.51) 

The  differences  observed  can 

be  attributed 

almost  entirely 

to  each  language’s  rule  for  the  placement 

of  semicolons. 

Twenty-seven 

of  the  96  errors  in  TOPPS  programs  and 

12  of 

the  81  errors 

in  TOPPSII  programs  involved 

semicolons. 

Of 

the  TOPPSII 

errors,  10  resulted  from  extra 

semicolons : 

one 

after  the  selection  expression 

in  an  if  statement  and 

the 

other  nine 

as  null  statements 

in  else  clauses.  With 

these 

errors  removed,  only  the  difference  in  errors  is  mildly 

s ignif ica  nt. 

TOPPS 

TOPPSII 

Level 

Errors 

6.90(14.65) 

4 .60  (1  1.90) 

<20% 

Occurrences 

31.70  (13,70) 

22.87(12.53) 

Error  runs 

12.80 (12.  15) 

15. 13 (13. 57) 

Persistence 

4.59(70.04) 

4.97 (68.96) 

The  TOPPS  subjects  had  more  difficulty  than  the  TOPPSII 
subjects  with  the  requirement  that  each  if  construct  must 
contain  an  else  clause^  and  with  bracketing  compound 
expressions  in  the  then  and  else  clauses.  For  errors  caused 
by  omission  of  else  clauses,  all  the  figures  favor  the 
TOPPSII  group. 
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T0PP5 


TOPPSII 


Level 


Errors 
Occurrences 
Error  runs 
Persistence 
There  were 
persistence 
bracketing 
the  TOPPSII 


^  .90(15.35) 
2.80(15.35) 
1.10(15.20) 
1 .47  (12.60) 


.  33  (1  1.43) 
.47  (11.43) 
.27  (1  1.53) 
1.40(12.10) 
errors. 


<10% 

<10% 

<20% 

average 
and  four 
in 


bracketing 

in  the  TOPPS 
with  a 


SIX 

of  2. 00  runs, 
errors,  each 
programs. 


with 

programs 
persistence  of  one  run 


The  TOPPSII  subjects  had  more  difficulty  than  the  TOPPS 
subjects  with  a  different  error:  an  action  being  performed 
in  only  one  of  the  clauses  of  an  if  construct  nested  within 
a  repeat  body,  without  a  complementary  action  being 
performed  in  the  other  clause.  Eor  example,  the  value  of  a 
loop  control  variable  might  be  incremented  in  one  of  the 
clauses  of  an  if  construct,  but  not  altered  in  the  other 
clause.  In  the  following  example,  the  value  of  the  loop 
control  variable  (I)  is  not  altered  in  the  then  clause  of 
rhe  outermost  if  construct.  When  Hand  (I)  and  Hand  (5)  had 
the  same  values,  an  infinite  loop  resulted. 

T  •=  1  • 

—  •  1  9 

repeat ; 

if  Hand  (I)  =  Hand  (5)  then 

if  I  <  3  then  Pair2”T=  I; 

else  Kinds  ;=  I; 

end; 

e  1^  1  1  1 ; 

end; 

until  I  >  3 ; 
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six  of  these  errors,  with  an  average  persistence  of  7.43 
runs,  appeared  in  TOPPSII  programs,  but  only  a  single  error, 
which  persisted  fcr  five  runs,  occurred  in  TOPPS  programs. 


Both  groups  of  subjects  committed  errors  by  failing  to 

protect  the  updating  of  shared  data  with  a  critical  region. 

However,  even  some  of  the  subjects  who  protected  the 

updating  operation  failed  to  protect  tests  of  shared  data 

(e.g. ,  Buffer_size) . 

variable  Buffer^size; 

reusable  Buff er^contr oiler  with  1; 


parallel  procedure  In put_ spooler  ...; 


if  Buffer^size  >  4  then 

request (Buff er  controller) ; 
request  (Buffer)  ; 

Buffer_size  :=  Buffer^size  -  1; 

• 

release  (Buff er  controller)  ; 
else  end; 


end; 

The  case  statement  was  the  source  of  three  errors.  One 
error  resulted  from  omitting  a  label  on  a  clause  and  another 
from  having  labels  of  different  types  in  the  same  case 
statement.  The  most  severe  error,  lasting  for  nine  runs, 
was  the  result  of  a  poor  combination  of  language  design 
decisions.  In  TOPPSII,  multiple  labels  on  a  single  case 
clause  are  separated  by  commas,  and  a  colon  follows  the 
final  label.  Also,  the  null  statement  need  not  be  indicated 
by  a  semicolon.  Therefore,  when  a  subject  separated 
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multiple  labels  with  a  colon  instead  of  a  comma,  no  warning 

message  could  be  issued.  The  colon  was  treated  as  the  start 

of  a  new  clause  containing  no  statements. 

case  Process_no  of 
PO  :  .  . . ; 

P1:P2:  ...; 

P3 ;  .  . . ; 
end; 


5 . 4 . 7 . 3  Iteration  statements 


Augmenting  the  repeat  statement  with  the  _^r  statement 
was  expected  to  reduce  the  number  and  severity  of  errors 
committed  by  the  users  of  TOPPSII.  While  all  the  figures 
favor  TOPPSII,  only  the  difference  in  the  number  of  errors 
is  mildly  significant. 


Errors 
Occurrences 
Error  runs 
Persistence 
After  removing 
the  difference 


TOPPS 

7.00  (14.60) 
32.80  (14.  05) 
15.00  (13.55) 
4.69  (60.28) 


TOPPSII 
3.46(11.93) 
16.  00  (12.30) 
11.53(12.63) 
4.62(63.14) 


Level 

<20% 


the  errors  attributable  to  semicolons. 


in  persistences  is  significant. 


only 


Errors 
Occurrences 
Error  runs 
Persistence 


TOPPS 

3.70 (12.45) 
27.30 (13. 65) 
14.20  (13.05) 
7.38(49.80) 


TOPPSII  Level 

3.27(13.37) 

1  5.  80  (12.57) 

1  1.47  (12. 97) 

4.57  (38.74)  <5% 


The  principal  cause  of  iteration  errors  in  TOPPS 
programs  (other  than  semicolon  placement)  was  the  failure  to 
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use  a  repeat  expression  to  keep  a  process  active.  A  process 


terminates  when  it  executes  its  last  instruction.  Subjects^ 
evidently  expecting  a  process  to  continue  execution 
indefinitely,  often  failed  to  use  a  repeat  construct  to 
insure  the  continued  execution  of  the  process.  This  error 
was  observed  five  times,  with  an  average  persistence  of  5.20 
runs,  in  TOPPS  programs;  but  no  instances  of  the  error 
appeared  in  TOPPSTI  programs.  The  major  cause  of  errors  in 
TOPPSII  programs  was  incorrect  specification  of  the  exit 
condition  due  to  zero-indexing  on  character  strings  and 
arrays.  (This  error  is  discussed  in  more  detail  in  Section 
5.6.)  Ten  instances  of  this  error,  with  an  average 
persistence  of  4.70  runs,  were  observed  in  TOPPSII  programs, 
and  only  two  instances,  with  an  average  persistence  of  1.50 
runs,  were  observed  in  TOPPS  programs. 


Two  other  problems  with  exit  conditions  were  prominent 
in  the  subjects®  programs.  The  first  problem  involved 
testing  the  end  of  file  condition.  This  condition  becomes 
true  when  the  data  request  of  an  input  operation  is  not 
satisfied.  Subjects  would  often  use  the  value  of  the  data 
before  determining  if  any  data  was  available.  In  the 
following  example,  a  subject  wanted  to  read  a  card  and 
release  a  unit  of  the  resource  Jobs  containing  the  value  on 
the  card. 
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repeat 


Eof  ;=  input (Card  in) ; 
release (JobSr Card  in) ; 
until  -lEof; 

The  program  should  have  been  constructed  in  the  following 
manner: 

Eof  :=  input (Card  in)  ; 
if  -lEof  then 
repeat 

« 

release (Jobs^ Card  in) ; 

Eof  :=  in£ut  (card_in)  ; 
until  -tEof 

else  0 

Errors  of  this  type  appeared  once,  for  thirty  runs,  in  TOPPS 
programs  and  twice,  with  an  average  persistence  of  15.00 
runs,  in  TOPPSII  programs. 

The  second  error  was  characteristic  of  rhe  Spooling 
problem.  An  output  spooler  could  terminate  processing  only 
after  both  central  processing  units  had  finished.  Some 
subjects  in  each  group  constructed  output  spoolers  that 
terminated  after  receiving  a  message  from  only  one  of  the 
central  processing  units  (e.g.,  a  unit  of  the  resource  Lines 
containing  the  value  *0»)  ,  instead  of  messages  from  both  of 
these  processes. 

program  Op spool  ...  is 

repeat 

Line  :=  reguest (Lines) : 

until  Line  =  *0* 

end; 


120 


This  error  was  observed  twice  with  an  average  persistence  of 
13.50  runs  in  TOPPS  programs,  and  twice  with  an  average 
persistence  of  11.00  runs  in  TOPPSIT  programs. 

The  introduction  of  the  for  statement  resulted  in  only 
two  syntactic  errors:  a  missing  semicolon  at  the  end  of  a 
for  clause  and  a  missing  end.  As  was  previously  stated, 
subjects  were  apparently  reluctant  to  use  the  for  statement 
because  no  provision  was  made  to  escape  from  the  body  of  zhe 
statement  when  a  condition  was  satisfied.  The  major  use  of 
the  statement  was  to  initialize  arrays.  Thus,  some  errors, 
like  those  shown  in  the  TOPPS  programs  below,  were  probably 
averted.  The  following  program  was  intended  to  initialize 
the  elements  of  the  arrays  Alloc  and  Maxdemand,  but  the 
subscript  expression  exceeded  the  declared  bounds  for  the 
arrays . 

array  Alloc  bound  1; 

array  Maxdemand  bound  1; 

• 

J  :=  0; 

repeat 

Maxdemand  (J)  i-  0; 

Alloc  (J)  : =  0 : 

J  ;=  J  1; 

until  J  >  2; 

A  more  subtle  error  occurred  in  a  program  to  find  any 
matches  among  the  elements  numbered  one  to  five  of  the  array 
Card.  In  the  following  example,  the  last  element.  Card (5), 
was  never  examined. 
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array  Card  bound  5; 


A  ;=  1; 
repeat 

.  code  to  find  the  matches 
.  in  Card  (A) 

A  :=  A  ♦  1; 
until  A  =  5; 


5 . 5  Other  errors 


It  was  expected  that  the  errors  that  were  not 
attributable  to  any  of  the  design  decisions  would  be  about 
the  same  in  the  two  languages.  The  following  figures 
summarize  these  errors.  The  differences  in  errors, 
occurrences,  and  error  runs  all  favor  the  TOPPS  group,  but 
only  the  number  of  errors  is  even  mildly  significant.  The 
difference  in  persistences  favors  the  TOPPSII  group  and  is 
significant. 


Errors 
Occurrences 
Error  runs 
Persistence* 


TOPPS 

22.50 (10.45) 
77,00(12.70) 
24.20  (12.65) 
3.42 


TOPPSII  Level 

28.  47(14.70)  <20% 

81.27  (13.20) 

25.  20  (13.23) 

2.85  <10% 


5. 5. 1  Parameters 


In  both  languages,  parameters  are  passed  by  reference  to 
procedures  invoked  sequentially,  and  by  value  to  procedures 
invoked  concurrently.  This  mechanism  was  designed  to  avoid 
errors  resulting  from  race  conditions  (i.e.,  the  value  of  a 
parameter  passed  to  a  process  being  altered  by  the  calling 


procedure  before  the  called  procedure  accesses  the 
parameter)  . 


Two  errors  are  attributable  to  this  parameter  passing 
rule.  The  first  of  these  errors,  involving  the  inadvertent 
alteration  of  the  value  of  a  parameter  passed  by  reference 
(Buf f er_reguest)  ,  is  illustrated  in  the  example  below» 
array  Buf f€r_ request  bound  1; 


function  Safe_to_read_in__new_card  of  Br  . . .  ; 


Sort  knows  Br; 
variable  Dummy; 
if^BrCOr  >  Br(1)  then 
Dummy  :  =  Br  (0)  ; 

Br  (0)  :=  Br  (1)  ; 

Er(1)  ;=  Dummy; 

else  end ; 

§nd; 


call  Sort ; 


if  Saf €__to_read_in__new_ca rd  (Buf fer_request)  then  ... 

The  complementary  error  results  from  altering  the  value  of  a 
parameter  passed  by  value  (Geronimo) ,  in  the  expectation  of 
returning  a  value. 

program  Play  of  Geronimo  ...  is 


repeat 

if  Geronimo  =  1  then  ... 


until  Nsame  =  4; 
request  (Spoon); 
Geronimo  ;=  1; 
end; 
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5.4.2  Inheritance  of  environment :  resources 


A.S  is  the  case  with  variables  and  arrays,  resources 
suffer  from  errors  caused  by  the  automatic  inheritance  of 
environment.  This  problem  is  especially  severe  because 
resources  are  meant  to  be  used  to  communicate  between 
processes.  In  the  following  example,  Inpu t_spooler  and  Cpu 
were  to  communicate  through  the  global  resource 
?ull_in_buffer ,  but  its  inadvertent  redeclaration  within 
Input_spooler  prevented  the  desired  communication  from 
occurring . 

consumable  Full_in_buff er ,  ...; 

parallel  procedure  I nput_ spooler; 
consumable  Full_in_buff er ; 

repeat; 

# 

if  end _pf ilg  then 

release  (Full_in_buff er ,  Eof_flag) ; 

else  release  (Full_in_buf f er ,  Message) ; 

end ; 

until  end  of  file : 

end; 

procedure  Cpu  . , . ; 
repeat; 

In_string  ;=  reguest  (Full__in_buffer)  ; 

Until  In_string  =  Eof_indicator ; 

end; 
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5.5.3  Critical  sections 


In  both  languages,  critical  sections  are  implemented  by 
first  reguesting  a  unit  of  a  reusable  resource,  then 
executing  the  code  within  the  section,  and  finally  releasing 
the  unit  of  the  resource.  Errors  result  when  either  of  the 
request  or  release  operations  are  interchanged  or  improperly 
nested  in  other  constructs,  or  one  of  the  operations  is 
missing.  In  the  following  example,  either  the  repeat 
statement  must  be  within  the  two  operations,  or  both 
operations  must  be  within  the  body  of  the  repeat  statement, 
reusable  Buff er_contr oiler  with  1; 

« 

repeat ; 

request (Buff er_controller)  ; 

until  First_byte  =  I; 
release  (Buff er_contr oiler) ; 

In  the  next  example,  when  Bufsize  was  less  than  or  equal  to 
one,  the  unit  of  the  resource  was  not  released, 
reusable  Mutex  with  1 ; 

♦ 

repeat ; 

request  (Mutex) ; 
if  Bufsize  >  1  then 

9 

9 

release  (Mutex)  ; 
else  end; 

until  Job_card  =  Finish; 

Finally,  no  release  operation  appears  in  rhe  following 
example : 
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reusable  Mutex  with  1 ; 


repeat; 


if  Flag  =  1  then  request  (Mutex)  ; 
else  end; 


until  Job  =  Foj; 


Since  request  and  release  are 
provided  for  exchanging  messages  with 
no  restrictions  may  be  placed  on  their 
that  units  of  a  reusable  resource, 
subsequently  relinquished. 


rhe  same  operations 
consumable  resources, 
appearances  to  ensure 
once  obtained,  are 


Further  errors  result  from  failing  to  provide  mutual 

exclusion  for  operations  that  should  be  uninterruptable.  In 

the  following  example,  a  subject  wanted  to  count  (using 

Eufsize)  the  number  of  available  units  of  a  resource 

(Buffer) ,  but  failed  to  make  the  acquisition  of  the  resource 

and  its  counting  an  uninterruptable  operation. 

request  (Buffer)  ; 
request  (Mutex) ; 

Bufsize  :=  Bufsize  -  1; 
release  (Mutex)  ; 


5.5.4  References  to  resources 


Confusion  about  the  difference  between  the  name  and 
value  of  a  resource  manifested  itself  in  the  subjects* 
programs.  The  name  of  a  resource  is  a  reference  to  the 
available  units  of  a  resource.  The  value  of  a  unit  of  a 
resource  is  assigned  by  a  release  operation  and  delivered  by 


126 


it  is 


^  request  operation.  In  order  to  retain  this  value, 
necessary  to  assign  the  value  to  a  variable  or  array 
element.  Some  subjects  sought  to  access  the  value  of  a  unit 
of  a  resource  by  using  the  name  of  the  resource  (e.g.,  as  in 
the  first  argument  of  the  second  subst r  function  in  the 
following  program) . 

request  (Input_pile) ; 

repeat 

until  siibstr  (Hand  (Hand__nuinber ,  I)  ,  0, 1)  -•= 
substr  (Input_pile, 0, 1) ; 

5.5.5  Invocation  of  function  procedures 

Two  problems  occurred  in  invoking  procedures  as 
functions.  The  first  of  these  problems  was  caused  by  the 
requirement  that  the  invocation  of  a  procedure  accepting  no 
parameters  must  be  followed  by  the  empty  parameter  list 
(i.e. t  opening  and  closing  parentheses)  for  the  transfer  of 
control  to  occur.  The  second  error  involved  the 
unintentional  invocation  of  a  procedure  that  produced  a  side 
effect  (e.g..  Discard). 

if  Discard  (Card)  -•=  false  then 

release  (Out_pile,  Discard  (Card) )  ; 

output  (’player*.  Player,  ’picked  up®.  Card, 

*  and  discarded®.  Discard  (Card) )  ; 

5.5.6  Incrementing  array  elements 

The  use  of  the  addition  operator  to  increment  the  value 
of  simple  variables  was  discussed  as  an  example  of  useless 
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redundancy  in  Section  2.3.3.  One  subject  provided  an 
example  of  this  problem  when  dealing  with  array  elements. 
In  the  following  example,  a  counter  of  the  number  of  values 
in  an  array,  Toppile(O),  is  assigned  rhe  value  of 
Toppile(1),  rather  than  having  its  own  value  incremented, 
because  of  a  misplaced  parenthesis. 

if  Number  >  0  then  . . . 

else  Toppile  (Number)  :=  Toppile  (Number  <r  1); 

if  Toppile  (Number)  >  11  then  Toppile  (Number)  ;=  0; 
else  end; 

end; 

5 .  6  Errors  r  e^  t  e  d  to  subjects  *  backgrounds 

Several  types  of  errors  that  were  not  previously 
discussed  are  attributable  to  the  subjects*  backgrounds. 
The  most  prominent  of  these  errors  involved  zero-indexing  on 
character  strings  and  arrays.  Most  of  the  subjects  had 
prior  experience  with  some  language  that  used  one  as  the 
index  of  the  first  element  of  an  array  or  the  first  position 
in  a  character  string  (e.g.,  FOETSAN,  COBOL,  ?L/I,  or  API). 
It  proved  difficult  for  subjects  to  remember  that  the  length 
operation  did  not  return  the  position  of  the  last  character 
in  a  string.  In  the  following  example,  the  second  argument 
is  beyond  the  end  of  the  string. 

String_length  :=  length (A_ job) ; 

Index  :=  0; 

regeat 

An_output_line  :=  cat  (An_output_l ine, 

substr  (A  job,  Str ing__length-Index ,  1)); 

Index  :=  Index  +  1; 

until  Index  =  String_length ; 


128 


Subjects  also  used  the  value  one  to  select  the  first 
character  in  a  string. 

First_char  :=  substr (Card,  1,1); 

Errors  resulted  from  attempts  to  select  the  correct 
condition  to  terminate  a  repeat  construct.  Subjects  often 
realized  that  the  constant  in  the  termination  condition 
needed  to  be  greater  than  the  bound  of  the  array,  but  would 
select  the  wrong  operator  to  express  the  termination 
condition. 

No  significant  differences  for  the  error  measures  for 
these  types  of  errors  were  observed  between  the  two  groups. 
However,  the  two  TOPPSII  subjects  who  had  no  prior 
experience  with  any  high-level  language  (i.e..  Autocode 
experience  only)  made  no  errors  of  this  type. 

The  grammar  of  TOPPSII  was  constructed  to  disallow  the 
parenthesization  of  <logical  expression>s  in  order  to 
perform  some  syntactic  type  checking.  Two  subjects  with 
extensive  FORTRAN  experience  attempted  to  enclose  selection 
expressions  of  if  statements  in  parentheses,  as  in  the 
following  example. 

if  CRd_gueue_length  No_of_buff ers  -  6)  then  ... 

Note  also  that  the  subject  has  drawn  on  his  experience  with 
PL/I  in  replacing  the  relational  operator  >=  with  the 
illegal  relational  operator  “«<.  Another  subject  with  PL/I 
experience  attempted  an  array  assignment. 
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5 . 7  Compiler  and  interpreter  aids 


5.7.1  Compiler  aids 

Each  grammar  contained  one  production  to  repair  syntax 
errors.  A  production  was  inserted  in  the  TOPPS  grammar  to 
repair  errors  resulting  from  the  addition  of  a  semicolon  to 
the  last  expression  in  a  compound  expression.  Irs  semantics 
produced  an  error  message  and  returned  the  integer  value 
zero  as  the  value  of  the  expression.  This  production 
repaired  91  errors,  which  occurred  150  times  on  64  runs.  A 
production  was  included  in  the  TOPPSII  grammar  to  repair 
errors  caused  by  the  omission  of  a  semicolon.  This 
production  resulted  in  the  repair  of  14  of  the  32  errors 
attributable  to  semicolons  in  executable  statements  and  all 
six  of  the  errors  attributable  to  semicolons  in  declarations 
in  TOPPSII  programs.  The  14  errors  occurred  18  times  on  13 
runs,  and  the  6  errors  occurred  6  times  on  3  runs. 

The  TOPPSII  grammar  also  contains  productions  that 
discourage  the  use  of  arithmetic  operators  in  expressions 
containing  functions  that  return  character  or  logical 
values-  Attempting  to  subtract  one  character  value  from 
another  resulted  in  a  compile-time  diagnostic. 

No_of_lines  :=  substr  (Line_in,  0,  1)  -  *0*; 

Similarly,  an  attempt  to  subtract  the  value  returned  by  the 
all  function  produced  a  message  from  the  parser. 

if  Input_nbr  <  Bufsize  -  all  of  (...)  then  ... 
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The  TOPPSII  compiler  also  contained  an  implementation  of 


Morgan's  algorithm  for  the  repair  of  keypunching  and 
spelling  mistakes.  The  figures  below  summarize  all  the 
keypunching  and  spelling  mistakes  observed.  The  differences 
in  errors,  occurrences,  and  error  runs  all  favor  the  TOPPS 
group,  perhaps  reflecting  the  reliance  of  the  TOPPSII 
subjects  on  the  algorithm  to  locate  and  repair  (at  least  on 
the  current  run)  these  errors.  However,  none  of  these 
differences  is  statistically  significant.  The  difference  in 
persistences  favors  the  TOPPSII  subjects  and  is  mildly 
significant. 


Errors 
Occurrences 
Error  runs 
Persistence 


TOPPS 

2.40(11.50) 
3.20  (11.90) 
1 .80(1 1.05) 
1.33(50.83) 


TOPPSII 
4.33  (14.00) 
4.67(13.73) 
2.67(14.30) 
1.08(42.84) 


Level 


<20% 


In  TOPPSII,  16  of  the  20  errors  in  reserved  words,  3  of 
the  13  errors  in  the  declaration  of  identifiers,  and  20  of 
the  31  errors  in  references  to  identifiers  were  repaired 
successfully  using  this  algorithm.  Removing  these  errors 
and  examinining  only  the  unrepaired  errors  results  in  all 
the  error  measures  favoring  the  TOPPSII  group  with  the 
differences  in  occurrences  and  persistences  being  mildly 
significant. 
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E  rrors 
Occurrences 
Error  runs 
Persistence 


TOPPS 

2.40 (15. 20) 
3.20(15.55) 
1 .80 (14.80) 
1.33  (28.29) 


T0PP3II 
1 .73(11 .53) 
1.80  (1  1.30) 
1  .27(11.80) 
1.04  (22.92) 


Level 


<20% 


<20% 


5.7.2  Interpreter  aids 


Both  interpreters  have  diagnostic  facilities  for 
detecting  incorrect  numbers  of  parameters,  uninitialized 
variables,  and  subscript  values  out  of  the  declared  bounds 
of  arrays.  Since  the  classification  of  errors  in  this  study 
was  performed  by  primary  causes,  the  figures  below  do  not 
necessarily  represent  the  contribution  that  these  diagnostic 
tools  made  by  being  the  manifestations  of  other  types  of 
errors.  All  the  differences  in  errors  involving  the 
incorrect  number  of  parameters  favor  the  TOPPS  group.  The 
differences  in  occurrences  and  error  runs  are  mildly 
significant,  and  the  difference  in  persistences  is  highly 
significant. 


Errors 
Occurrences 
Error  runs 
Persistence 


TOPPS 

1 .50(12.  10) 
2.80(11.6  0) 
1.50(11.40) 
1 . 87  (  1 4 . 6  3) 


TOPPSII 
1.87(13.60) 
7.73(13.93) 
3.33(14.07) 
4.  14(25.95) 


Level 


<20% 

<20% 

<.5% 


The  differences  in  the  number  of  errors  and  occurrences 
attributable  to  uninitialized  variables  favor  -che  TOPPS 
group  and  are  mildly  significant.  The  difference  in 
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persistences 

favors  the 

TOPPSII  group. 

but  is 

significant. 

\ 

TOPPS 

TOPPSII 

Level 

Errors 

.  50(10.30) 

1  .47(14.80) 

<20% 

Occurrences 

2.80(11.00) 

5.13(14.33) 

<20% 

Error  runs 

2.60(1  1.45) 

3.27(14.03) 

Persistence 

5.  60(16.80) 

3.50(13.36) 

Too  few  errors  whose  primary  cause  was  subscript  out  of 
the  declared  bounds  of  an  array  were  discovered  to  make  any 
meaningful  comparisons. 

5.8  Summary  of  results 

The  following  table  summarizes  the  statistically 
significant  results  obtained  in  the  evaluation  of  the 
hypotheses  of  this  study.  Direction  indicates  the  language 
with  the  smaller  measure. 
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Hypothesis 

Measure 

Direction 

Level 

All  errors 

Occurrences 

TOPPSII 

<20% 

All  changed  features 

Errors 

TOPPSII 

<5% 

Occurrences 

TOPPSII 

<5% 

Semicolons 

Executable  stmts 

Errors 

TOPPSII 

<.5% 

Occurrences 

TOPPSII 

<•5% 

Error  runs 

TOPPSII 

<.5% 

Declarations 

Errors 

TOPPSII 

<5% 

Occurrences 

TOPPSII 

<5% 

Error  runs 

TOPPSII 

<5% 

Expression  evaluation 

Errors 

TOPPSII 

<5% 

Occurrences 

TOPPSII 

<5% 

Error  runs 

TOPPSII 

<5% 

Relation -connectors 

Persistence 

TOPPSII 

<20% 

Assignment 

Error  runs 

TOPPSII 

<20% 

Persistence 

TOPPSII 

<.  1% 

Environment  inheritance 

Syntax  errors 

Errors 

TOPPS 

<5% 

included 

Occurrences 

TOPPS 

<10% 

Error  runs 

TOPPS 

<10% 

Persistence 

TOPPSII 

<•5% 

Syntax  errors 

Persistence 

TOPPSII 

<20% 

excluded 

Statement  brackets 

Errors 

TOPPSII 

<5% 

Occurrences 

TOPPSII 

<5% 

Error  runs 

TOPPSII 

<5% 

Persistence 

TOPPSII 

<10% 

Constants 

Persistence 

TOPPSII 

<5% 

Selection  statements 

Errors 

TOPPSII 

<10% 

Occurrences 

TOPPSII 

<20% 

Iteration  statements 

Errors 

TOPPSII 

<20% 
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CKAPTEP.  SIX 


CONCLUSIONS  AND  DIRECTIONS  FOR  RESEARCH 

6  ^  Ir.trodiiction 

The  contributions  of  this  thesis  toward  the 
identification  of  language  features  that  enhance  reliability 
appear  in  Section  1.2.  Chapter  Two  presents  general  design 
principles  and  specific  language  features  that  may  enhance 
reliability.  Chapter  Four  contains  an  application  of  some 
of  these  principles  and  features  in  the  redesign  of  a 
programming  language.  Chapter  Three  describes  a  methodology 
to  gather  empirical  evidence  with  which  to  support  or 
discredit  language  design  decisions.  Chapter  Five  discusses 
the  results  of  an  experiment  that  measured  the  effect  of 
several  language  design  decisions  in  one  environment. 

This  chapter  outlines  the  limitations  of  the  methodology 
and  results.  Next,  the  implications  of  the  results  for 
language  design  are  discussed.  Finally,  directions  for 
research  are  presented. 

6 . 2  Limi^^on s  of  the  methodology  and  results 

In  Chapter  Three,  the  methodology  for  conducting  this 
study  was  described  and  criticized.  Some  of  the 
shortcomings  are  common  to  much  experimental  work  {e.g.,  the 
quality  of  the  subjects  and  che  amount  of  data  obtained) . 
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It  was  expected  that  selecting  subjects  from  a  fourth  year 
course  in  a  computer  science  specialist  program  would 
provide  relatively  sophisticated  programmers.  However,  the 
average  number  of  errors  committed  by  the  subjects  is  so 
large  that  it  raises  doubts  about  this  assumption.  Some  of 
the  errors  might  be  due  to  the  fact  that  the  subjects  were 
learning  a  new  language,  or  that  the  subjects  were  making 
their  first  attempt  at  concurrent  programming. 

Making  a  final  decision,  based  upon  a  small  number  of 
errors,  that  one  form  of  a  language  feature  is  superior  to 
another  would  be  a  mistake.  However,  large  differences 
among  even  a  small  number  of  errors  should  convince  a 
language  designer  that  further  study  is  necessary.  The 
passivity  of  this  type  of  study,  observing  whatever  errors 
subjects  happen  to  make,  is  a  drawback  in  this  situation. 
There  is  no  way  to  ensure  that  subjects  make  the  type  of 
errors  one  wishes  to  examine,  although  proper  problem 
selection  may  make  certain  errors  slightly  more  likely  to 
occur.  For  example,  selecting  problems  that  required 
complicated  arithmetic  expressions  would  probably  increase 
the  number  of  errors  attributable  to  the  manner  in  which 
expressions  are  evaluated. 

Another  problem  with  this  study  is  the  subjectivity  with 
which  the  errors  were  classified.  This  process  was 
performed  by  the  experimenter  alone.  Youngs  proposed  a 
solution  to  this  problem.  Three  experienced  programmers 
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were  hired  to  independently  classify  the  same  body  of  data^, 
and  the  proportion  of  unanimous  decisions  in  classifying 
errors  was  then  recorded. 

The  decision  to  study  simultaneously  several  language 
features  led  to  several  complications.  For  example^  the 
decisions  to  introduce  global  named  constants  and  to  prevent 
the  automatic  inheritance  of  environment  in  TOPPSII  resulted 
in  errors  when  attempts  were  made  to  include  named  constants 
in  nonlocal  variable  lists.  If  only  one  of  these  design 
decisions  were  made,  these  errors  would  not  have  appeared. 
However,  language  features  must  be  evaluated  in  the  context 
in  which  they  are  to  be  used,  and  creating  too  small  a 
micro-language  may  prevent  observation  of  errors  resulting 
from  the  interaction  of  language  features. 

Perhaps  the  most  serious  limitation  of  this  study  is  its 
treatment  of  two  effects:  learning  a  language  and  the  size 
of  the  problems.  The  need  to  avoid  bias  introduced  by 
subjects*  prior  experiences  with  programming  languages  led 
to  the  use  of  languages  with  which  Eone  of  the  subjects  were 
familiar.  In  order  to  find  all  the  errors  in  a  program, 
relatively  small  problems  were  examined. 

The  greater  number  of  errors  in  TOPPS  programs  observed 
in  this  study  belied  the  the  expectation  that  the  greater 
syntactic  complexity  of  TOPPSII  would  make  it  more  difficult 
to  learn  than  TOPPS.  There  are  some  possible  explanations 
for  this  effect.  In  Section  5.4.7,  figures  on  statement 
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usage  suggest  that  the  TOPPSII  subjects  avoided  iche  use  of 
some  of  the  more  complex  features  of  the  language  (e.g.,  the 
case  and  for  statements) .  Another  reason  for  the  apparent 
ease  with  which  subjects  learned  TOPPSII  may  be  the 
similarity  of  its  features  to  those  cf  ether  programming 
languages  and  mathematics. 

A  crude  measure  of  the  learning  effect  can  be  observed 
in  the  following  tables.  The  columns  indicate  the 
percentage  of  relevant  errors,  relarive  to  the  total  number 
of  errors,  that  occurred  in  the  first  and  second  halves  of 
program  development  (measured  by  the  number  of  runs) 
respectively.  The  division  of  errors  that  occurred  in  the 
first  half  of  the  runs  compared  to  those  that  occurred  in 
the  second  half  was  511-167  in  TOPPS  and  539-200  in  TOPPSII. 
(The  figures  for  errors  in  the  second  half  of  the  runs 
contain  both  errors  that  occurred  in  the  first  half  of  the 
runs  and  persisted  until  the  second  half  of  the  runs,  and 
errors  that  occurred  only  in  the  second  half  of  the  runs.) 

TOPPS 


Error 

First 

Second 

Semicolon 

Declarations 

4.  5% 

1 .8% 

Executable  statements 

36.  8% 

12.  6% 

Expression  evaluation 

1.055 

1.2% 

Boolean  operators 

0.  4% 

0.6% 

Assignment 

3.7% 

8.  4% 

Statement  brackets 

2.  3% 

1.2% 

Inheritance  of  environment 

0.  8% 

1.8% 

Other  errors 

50.5% 

ioo7o% 

72.5% 

iooTi% 
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TOPPSII 

Error 

First 

Second 

Semicolon 

Declarations 

1.1% 

0.0% 

Executable  statements 

5.6% 

1.0% 

Expression  evaluation 

0.  0% 

0.0% 

Relation- connectors 

0.  4% 

0.0% 

Assignment 

1.9% 

1.0% 

Statement  brackets 

0.9% 

1.0% 

Inheritance  of  environment 

3.9% 

4.0% 

Other  errors 

86.3% 

9  3.0% 

100.1% 

100.0% 

The  interesting  errors  are  those  whose  percentages  increased 
or  remained  large  during  the  second  half  of  program 
development  despite  the  nature  of  the  runs  during  this 
period  (e.g.,  polishing  output  or  adding  comments).  These 
errors  are  much  more  difficult  for  subjects  to  locate  than 
those  that  violate  the  syntax  of  the  language.  Thus,  in 
some  sense,  these  features  are  more  difficult  for  subjects 
to  master. 


Dijkstra  argues  that  language  design  decisions  that  rely 

on  the  previous  experience  of  subjects  merely  replace  what 

is  "convenient”  with  what  is  "conventional”  [Dijkstra  1974]. 

”Today®s  programmers  have  a  past  and  it  seems  legitimate 
to  try  to  study  the  adequacy  of  that  past,  ...  Even  if 
that  influence  is  found,  it  remains  for  me  still  an  open 
question,  whether  the  way  in  which  the  programming  task 
is  presented  should  adapt  to  that  situation,  or  whether 
we  should  try  to  give  our  programmers  «a  more  adequate 
past*,  i.e.,  try  to  educate  them.” 

This  argument  is  certainly  persuasive  when  language  design 

decisions  are  based  solely  on  subjects*  prior  experience 

with  programming  languages  (e.g.,  semicolon  placement). 

However,  when  decisions  are  based  on  more  general  experience 
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(e.g. ,  expression  evaluation  in  mathematics) ,  there  seems 
little  chance  that  subjects  can  be  provided  with  "a  more 
adequate  past.”  Even  some  decisions  based  principally  on 
prior  experience  with  programming  languages  may  prove 
beneficial  if  other  considerations  come  into  play  (e.g.,  the 
similarity  of  the  symbols  for  assignment  and  comparison 
leading  to  inhibition) . 

The  second  problem  with  the  study  was  the  size  of  the 
programming  tasks.  Some  errors,  especially  those  caused  by 
violations  of  the  syntax  of  the  programming  language,  would 
probably  be  no  more  severe  in  larger  programs.  The 
preceding  tables  indicate  that  such  errors  (e.g.,  those 
involving  semicolons)  diminish  in  frequency  during  the 
second  half  of  program  development.  However,  other  errors 
such  as  those  involving  the  inheritance  of  environment  do 
not  diminish  in  frequency.  Furthermore,  large  systems  with 
several  programmers  are  likely  to  increase  the  severity  of 
errors  of  these  types. 

6 . 3  Implications  for  language  design 

The  results  of  this  thesis  demonstrate  that,  at  least  in 
one  environment,  several  language  design  decisions  affect 
reliability  significantly.  TOPPS  had  been  used 
"satisfactorily”  for  several  years,  and  each  of  its  bad 
features  is  shared  with  other,  more  widely-used  languages. 
Yet  a  few  simple  changes  produced  striking  results. 
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In  using  the  semicolon  as  an  expression  separatory 
rather  than  as  a  statement  terminatcri,  TOPPS  was  following  a 
long  and  honorable  tradition  (ALGOL  60,  Pascal,  BLISS, 
etc.) .  However,  the  TOPPSII  form  (similar  to  that  of  PL/I) , 
led  to  an  order  of  magnitude  reduction  in  the  number  of 
semicolon  errors.  Of  course,  semicolon  errors  are  rather 
trivial.  (They  generally  do  not  persist  more  than  one 
run.)  However,  a  small  modification  to  the  language  would 
have  eliminated  errors  that  occurred  on  more  than  a  quarter 
of  all  compilations.  It  is  interesting  to  note  that  over 
14%  of  the  errors  occurring  in  the  second  half  of  program 
development  in  TOPPS  are  attributable  to  semicolons 
(compared  to  1%  for  TOPPSII) . 

Part  of  the  difference  in  the  number  of  semicolon  errors 
can  be  explained  by  the  subjects*  prior  experiences  with 
statement-oriented  languages.  The  implications  of  designing 
programming  languages  to  profit  from  programmers®  prior 
experiences  are  by  no  means  clear.  It  would  be  interesting 
to  conduct  similar  observations  with  subjects  who  were 
familiar  with  only  expression-oriented  languages. 

While  the  desirability  of  using  programmers*  backgrounds 
as  a  language  design  criterion  may  be  in  doubt,  the 
desirability  of  uniformity  is  clear.  The  uniformity  of  both 
the  semicolon  rules  for  declarations  and  executable 
statements,  and  the  bracketing  structure  of  TOPPSII  led  to 
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fewer  and  less  severe  errors  attributable  to  these  language 
features* 

At  the  other  end  of  the  scale  from  these  trivial  errors 
are  four  classes  of  infrequent  errors  with  very  high 
persistence:  assignment  errors,  inheritance  of  environment 
errors,  expression  evaluation  errors,  and  relation-connector 
errors.  The  persistence  of  each  of  these  classes  of  errors 
in  TOPPS  was  about  half  the  total  number  of  runs  needed  to 
complete  a  program.  It  is  reasonable  to  assume  that  these 
errors  would  be  even  more  persistent  in  larger  programs, 
adding  even  greater  weight  to  the  already  significant 
improvements  made  by  TOPPSII.  Furthermore,  the  relative 
frequencies  of  these  four  classes  of  errors  in  TOPPS 
approximately  doubled  in  the  second  half,  making  it  seem 
unlikely  that  they  are  solely  due  to  unfamiliarity. 

The  persistence  of  assignment  errors  in  TOPPS  calls  into 
serious  question  the  treatment  of  the  assignment  symbol  as 
’’just  another  operator."  Expression-oriented  languages 
using  this  convention  (e.g. ,  ALGOL  68)  may  cause  unsuspecred 
reliability  problems.  Other  expression-oriented  languages 
using  an  assignment  operator  quite  different  from  =  (e.g., 
<-  in  APL  and  BLISS)  probably  avoid  some  of  these  errors. 
However,  these  languages  provide  no  better  error  detection 
for  substitutions  of  =  for  <-  than  languages  using  :=  as  the 
assignment  symbol.  Certainly,  an  observation  of  subjects 
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programming  in  a  language  which  used  <-  as  the  assignment 
symbol  would  be  interesting. 

The  TOPPSII  restrictions  on  inheritance  of  environment 
attempted  to  eliminate  some  very  persistent  errors,  at  the 
cost  of  introducing  a  few  trivial  ones.  iJhile  the 
comparison  of  errors  that  were  not  detectable  syntactically 
did  favor  the  TOPPSII  group,  the  figures  were  disappointing. 
The  scope  rules  of  TOPPSII  still  permit  subtle  errors  to  be 
made,  even  though  a  programmer  must  make  his  interface 
decisions  explicit.  Examining  the  errors  that  occurred  in 
TOPPSII  programs  leads  to  the  conclusion  that  more  redundant 
information  would  be  helpful  (e.g.,  access  rights  in  the 
declarations  specifying  both  which  data  items  must  be 
inherited,  and  which  data  items  can  not  be  inherited). 
Perhaps  it  would  be  worthwhile  to  warn  a  programmer  that  he 
had  redeclared  a  global  name,  and  that  any  assignments  to 
the  identifier  would  only  be  communicated  to  inner 
procedures. 

The  expression  evaluation  rules  of  TOPPS  are  similar  to 
those  of  API.  Only  two  programmers  (one  in  each  group)  had 
previously  programmed  in  API,  while  all  but  one  had 
experience  using  some  language  (not  to  mention  mathematics) 
with  left-tO“”right  evaluation  of  expressions  and  traditional 
operator  precedence.  Thus  the  greater  frequency  of  errors 
in  TOPPS  may  be  at  least  partially  explained  in  terms  of 
prior  experience.  However,  the  high  persistence  of  these 
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errors  seems  incompatible  with  the  claims  for  the  benefits 
of  "naturalness”  sometimes  made  for  the  APL  rules. 

Similarly,  errors  involving  infix  relation-connectors 
seem  to  be  difficult  to  find  and  remove.  However,  the  small 
number  of  these  errors  observed  makes  any  conclusions  rather 
tenuous. 

6 ^  Directions  for  research 

Further  analysis  of  the  data  gathered  in  this  study 
would  be  interesting.  For  example,  it  would  be  interesting 
to  plot  the  number  of  errors  existing  in  versions  of  a 
program  against  the  run  number  of  the  program,  to  develop  a 
feeling  for  rhe  pattern  of  debugging.  Alsc,  comparing  the 
differences  in  the  errors  made  during  the  solution  to  the 
two  problems  would  determine  whether  or  not  problem 
selection  would  allow  errors  characteristic  of  certain 
constructs  to  be  observed. 

The  figures  presented  in  Section  6.2  indicate  success  in 
eliminating  errors  associated  with  six  of  the  specific 
language  features.  In  the  first  half  of  program 
development,  errors  attributable  to  these  features  accounted 
for  one-half  of  the  errors  in  TOPPS  programs,  but  only  one- 
seventh  of  the  errors  in  TOPPSII  programs.  The  question 
that  remains  is  how  much  can  be  done  with  the  remaining  half 
of  the  errors. 
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other  results  of  this  study  raise  farther  questions. 
Does  the  use  of  the  assignment  symbol  rather  than  ;=^ 
lead  to  the  production  of  more  reliable  software?  How  much 
additional  redundancy  is  helpful  in  the  explicit  statement 
of  interfaces?  Other  language  features  not  included  in  this 
study  could  also  be  examined.  Among  the  most  interesting  of 
these  features  are  selection  by  name  rather  than  by  position 
in  parameter  lists^  languages  with  compile-time  type 
checking  rather  than  either  languages  with  dynamic  typing  or 
”typeless”  languages,  multiple  exits  from  control  constructs 
rather  than  single  exits,  and  multi-level  exits  from  control 
constructs  rather  than  single- level  exits. 

The  observation  techniques  employed  in  this  study  could 
also  be  applied  to  the  study  of  programming  methodologies 
(e.g. ,  stepwise  refinement,  action  clusters,  modular 
decomposition,  etc.) .  One  group  of  subjects  could  be 
trained  to  program  in  one  methodology,  and  another  group  of 
subjects  in  a  second  methodology.  The  groups  could  then  be 
asked  to  solve  the  same  problems  with  the  same  languages. 

Finally,  some  investigation  of  the  effects  of  learning 
and  scale  on  the  programming  process  should  be  attempted. 
Observing  subjects  writing  programs,  first  with  a  new 
language  and  then  at  some  later  time  period  with  the  same 
language,  should  give  some  indication  of  the  learning 
effect.  This  would  provide  a  fairer  basis  for  comparison  of 
language  design  decisions  based  on  subjects®  backgrounds 
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(e.g.,  semicolon  placement  and  the  method  of  expression 
evaluation) .  Observing  subjects  writing  larger  programs 
should  provide  some  insight  into  the  problems  of  scale. 
Errors  with  large  persistences  (e.g.,  those  attributable  to 
inheritance  of  environment)  will  probably  be  more  severe  in 
the  larger  programs,  but  the  rate  of  growth  of  their 
severity  is  impossible  to  predict. 

There  is  no  reason  to  believe  that  languages  designed 


without 

explicit  concern 

for  reliability 

will 

be  suitable 

for  the 

production  of  reliable  programs. 

Each 

of  the  major 

programming  languages  has 

flaws  that  can 

be 

expected 

to 

reduce 

the  reliability 

of  programs. 

This 

thesis 

has 

surveyed 

many  alternative 

features  that 

might 

reduce 

the 

frequency  and/or  persistence  of  errors,  and  has  provided 
empirical  evidence  to  support  the  usefulness  of  some  of 
them.  However,  there  is  still  much  to  be  done,  both  in  the 
design  cf  languages  which  enhance  the  reliability  of 
programs,  and  in  the  experimental  evaluation  of  language 
design  decisions,  particularly  to  discover  the  effects  of 
different  environments. 
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CO^^PIJT^F  SYS'^EYS  PESS?\RCH  GPOIT? 

>«'  CSPG-I  EMFIPTCr.L  COMPA-PTSON  OF  LR(k)  AND  PPECFDENCE  ??iRSERS 
J.J.  Horning  and  W.P.  Lalonde,  SeptembRr  1970 
f.ACK  SIGPLAN  Notices,  November  1970] 


CSRG-2 


*  CSRG-3 


*  CSPG-4 

TSRG-5 


AN  FFFTCIFNT  LALP  PAPSEP  GENERATOR 
W.P.  Lalonde,  February  1971 
[M.A.Sc.  Thesis,  EE  1971] 

A  PPOCESSOR  GENFPATOR  SYSTEN 
J.D.  Gorrie,  February  1971 
[M.A.Sc.  Thesis,  F^  1971] 

^YLAN  nSFR»?  MANUAI 
P.E.  Eonzon,  March  1971 

DIAL  -  A  PROGRAMMING  SYSTT^M  FOP  INTERACTIVE  ALGEPBMC 
MAN  I PHI ATI ON 

Alan  C.M.  Frown  and  J.J.  Horning,  March  1971 


*  CSRG-O  ON  DEADLOCK  IN  COMPUTER  SYSTEMS 
Richard  C,  Holt,  April  1971 
[Ph.D.  Thesis,-  Dept,  of  Computer  Science, 
Cornel]  University,  1971] 


CSRG-7  THE  STAP-PING  SYSTEM  OF  LOOSELY  COUPLED  DIGITAL  DEVICES 
John  Neill  Thomas  Potvin,  August  19*71 
[M.A.Sc.  Thesis,  EF  1971] 

*  CSRG-P  FILE  OFGANI7ATION  AND  STPUCTUPF 
G.M.  S-^-acey,  August  1971 


CSRG-H  DESIGN  STUDY  FOP  ?-  TWO-DIMENSIONAL  COMPUTEK-ASS  IS  IFD 
ANIMATION  SYSTEM 
Kenneth  B.  Evans,  January  1972 
[M.Sc.  Thesis,  DCS  1972] 


CSRG-10  ROW  A  PROGRAMMING  LA^NGUAGE  IS  USED 

William  Gregg  Alexander,  February  19*72 
[M.Sc.  Thesis,  DCS  1971] 

CSRG-11  PPCJEC'^  SUE  STATUS  REPORT 

J.W.  Atwood  (ed.),  April  1972 


CSRG-12  THREE  niMENSIONA.L  DATA  DISPL.AY  WITH  HIDDEN  LINE  REtMOVAL 
Rupert  Pramall,  April  1972 
[M.Sc.  Thesis,  DCS  1971] 


+  Abbreviations: 

DCS  -  Department  of  Computer  Science,  Univarsity  of  Toronto 
FE  -  Depar'i'm  en*:  of  F  lect  rica  1  Engineer  in  g.  University  of 
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*  -  Out  of  prir.'*' 


CSRG-13  A  SYNTAX  DIF.r.CTFD  FFFOP  FFCOVEFY  METHOD 
Lewis  F.  James,  May  1D72 
[M.Sc.  Thesis,  DOS  1972] 

CSRG-14  THE  USE  OF  SERVICE  TIME  DISTRIBUTIONS  IN  SCHEDULING 
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[Ph.D.  Thesis,  CommiTtee  on  Information  Sciences, 
University  of  Chicago,  1971;  JACM,  January  1974] 

CSRG-15  PROCESS  STFUCTUP.ING 

J.J.  Horning  and  B.  Ran dell,  June  1972 
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CSRG-16  OPTIMAL  PKOCESSOF  SCHEDULING  WHEN  SERVICE  TIMES  ARE 

HYPFFFXPONENTIALLY  DISTFIBU'^FD  AND  PFEEMriON  OVERHEAD 

TS  NOT  NEGLIGIBLE 

Kenneth  C.  Sevcik,  June  1972 

[Proceedings  of  the  Symposium  on  Computer -Comma nicat ion , 
N-^^t works  and  “^ele traffic. 

Polytechnic  Institute  of  Brooklyn,  1972] 


CSRG-17  PROGRAMMING  LANGUAGE  TFANSLATION  TECHNIQUES 
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CSRG-18  A  COMPAFATIVE  ANALYSIS  OF  SEVERAL  DISK  SCHEDULING 
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C.J.M.  Turnbull,  Sept^^mber  1972 

CSRG-19  PROJEC'"  SUE  AS  A  LEARNING  EXPERIENCE 
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[Proceedings  AFIPS  Fall  Joint  Computer  Conference, 
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CSRG-20  A  STUDY  OF  LANGUAGF  DIFFCTED  COMPUTER  DESIGN 
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[Ph.D.  Thesis,  Computer  Science  Department, 
Stanford  University,  1972] 
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AN  IMPLEMENTATION  LANGUAGE  FOR  MINICOMPUTERS 
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CSRG-23  COMPILER  STRUCTURE 
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[Proceedings  of  the  USA-Japan  Computer  Conference,  1972] 
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ENGINEERING 
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THF  INVESTIG?^.TION  OF  SERVICE  TIME  DISTRIBUTIONS 
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PSYCnOLOGICAL  COMPLEXITY  OF  COMPUTER  PROGRAMS: 

AN  INI'^IAL  EXPERIMENT 
Larry  Weissnan,  August  1973 

STRUCTURED  SUBSETS  or  THE  PL/I  LANGUAGE 

Richard  C.  Holt  and  David  B.  Wortraan,  Ocrober  1973 

ON  THE  REDUCED  MAvTFIX  REPRESENTATION  OF  LR(k) 

PARSER  TABLES 
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[Ph.D.  Thesis,  EE  1973] 

A  STUDENT  PFOJSC'^  FOR  AN  OPERATING  SYSTEMS  COURSE 
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A  PSEUDO-MACHINE  FOP  CODS  GENEPzATION 
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SCHEDUJ.ING  MULTIPLE  RESOURCE  COMPU'^^R  SYSTEMS 
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F.  Lochovsky  and  D.  Tsichritzis,  May  1974 
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August  1974 
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COMPUTING  ENVIRONMENT 
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